System

Abstract

A system is provided.SOLUTION: This system includes a means for allowing a user to read document data such as a receipt or a certificate, a means for uploading the read data to a server, a means for analyzing the data by the server, and for generating information necessary for final declaration, a means for transmitting the generated information to the terminal of the user, and for allowing the user to confirm and correct the information, and a means for automatically generating a declaration based on the confirmed information, and for submitting it to an electronic declaration system.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The current tax return process is full of technical terms and complex steps, placing a significant burden on the elderly and those who are not adept at using information devices. Furthermore, existing AI tax return software is primarily aimed at businesses and sole proprietors, resulting in high costs and the hassle of contracts. This makes filing tax returns difficult for ordinary individual users. Furthermore, data entry errors and the hassle of manual organization remain common during filing, and effective methods to alleviate these issues are needed. [Means for solving the problem]

[0005] The tax return assistant system of the present invention includes a means for users to read document data such as receipts and certificates, and a means for uploading the read data to a server. The server includes a means for analyzing the data and automatically generating information necessary for tax returns, which is then sent to the user's terminal for confirmation and correction. The system also includes a means for automatically generating tax returns based on the confirmed information and submitting them to an electronic filing system. Furthermore, the system includes a means for using image processing technology to perform image recognition on the document data and a means for using natural language processing technology to analyze the extracted text data. This allows users to significantly reduce the burden of tedious manual input work and enables them to quickly file tax returns based on accurate data.

[0006] "Document data" refers to image data that includes information such as receipts and certificates that users need to file their tax returns.

[0007] A "terminal" is a device used by a user, such as a smartphone or computer.

[0008] The "server" is a computing system that analyzes uploaded document data and generates and transmits the information required for tax returns.

[0009] "Image processing technology" is a technique for extracting desired information from image data. It is done using libraries such as OpenCV.

[0010] "Natural language processing technology" is a technology for analyzing text data and extracting meaningful information.

[0011] "Automatic generation" means that a program generates documents or data without human intervention.

[0012] The "electronic filing system" is a public system for submitting tax returns online. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] The tax return assistant system of this invention is designed to allow individual users to easily file tax returns. The system mainly works by having the user read document data, upload it to a server, analyze and generate the data, have the user check and correct it, automatically generate a tax return, and finally submit it to the electronic filing system.

[0035] Technical configuration

[0036] 1. Acquisition of document data

[0037] The user takes a photo of the receipt or certificate using a device such as a smartphone or computer. This captures the document data, which is then temporarily stored in the local storage of the user's device.

[0038] 2. Uploading data

[0039] The user uploads the acquired document data to the server through a dedicated tax return assistant app. The device then sends the user-selected images to the server while maintaining appropriate security.

[0040] 3. Data Analysis

[0041] The server analyzes the received document data using image processing and natural language processing technologies. First, it performs image recognition and extracts the text information written on the document as text. This extracted text data is then analyzed using natural language processing technology to identify important information such as the amount and name of the local government.

[0042] 4. Information Generation and Verification

[0043] Based on the parsed data, the server generates the information necessary for filing tax returns. This information is sent to the terminal as structured data in JSON format or similar and displayed to the user. The user can then check the displayed information and manually correct or add anything necessary.

[0044] 5. Automatic generation and submission of tax returns

[0045] The server automatically generates a tax return based on the final confirmed data. The generated tax return is then verified internally and then submitted to the electronic filing system. This series of processes allows users to easily file accurate tax returns.

[0046] Specific examples

[0047] For example, let's say a user uses a receipt for hometown tax donations. First, they take a photo of the receipt with their smartphone camera and upload the image to the server via the app. The server analyzes the image and extracts the amount and name of the local government. The extracted data is sent to the user's device, where the user can confirm and correct it. Finally, the server generates a tax return based on this information and submits it to the electronic filing system. This completes the tax return.

[0048] In this way, the system of the present invention allows users to simplify cumbersome procedures and save time and effort.

[0049] The processing flow will be explained below.

[0050] Step 1: Scan your documents

[0051] The user uses the camera function to take a photo of the document data and save it on the device. The user then launches a dedicated app and selects the image they took.

[0052] Step 2: Upload your data

[0053] The device sends the document data to the server. During transmission, the data is transferred securely using security protocols such as SSL. The upload progress is displayed on the user's device.

[0054] Step 3: Data analysis (image recognition)

[0055] The server receives the uploaded image data. Next, it uses image processing technology such as OpenCV to extract text from the document image. Then it uses a character recognition engine such as PyTesseract to recognize the text in the image and obtain the text data.

[0056] Step 4: Data analysis (natural language processing)

[0057] The server uses natural language processing technology to analyze the text data it acquires and extracts the information necessary for filing tax returns (e.g., amount, date, name of local government). During this process, the AI ​​algorithm identifies specific patterns and keywords, and categorizes and organizes the information.

[0058] Step 5: Generate and send information

[0059] Based on the analysis results, the server generates data summarizing the information required for filing, and this data is sent to the device in a structured format such as JSON.

[0060] Step 6: Verify and correct information

[0061] The device displays the received data to the user. The user can check the displayed information and manually edit or add anything if necessary. Once edits are complete, the user taps the "Confirm" button.

[0062] Step 7: Auto-generate tax returns

[0063] The server automatically generates tax returns based on the verified information, and the generated returns are validated internally to ensure accuracy.

[0064] Step 8: Submit to the electronic filing system

[0065] The server submits the generated tax return to the official electronic tax return system. Once the submission is complete, the server sends a notification to the user that the return has been completed. The notification can be sent via email or in-app notification.

[0066] These steps allow users to accurately file their tax returns with minimal effort on their part.

[0067] Example 1

[0068] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0069] Accounting and tax return procedures are extremely complicated and require a great deal of time and effort, especially for individual users. As a result, many users struggle to manage the necessary documents and prepare accurate tax returns. Furthermore, manual data entry and correction increases the risk of human error. Therefore, there is a need for a system that simplifies tax return procedures and processes them accurately and quickly.

[0070] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0071] In this invention, the server includes: [means using image processing technology to perform image recognition of document data]; [means using natural language processing technology to analyze text data extracted from the image]; and [means using JSON format to structuring data]. This enables users to easily digitize document data such as receipts and certificates, convert that data into information necessary for tax returns, automatically generate tax returns accurately and quickly, and submit them to the electronic filing system.

[0072] "User" refers to a person who uses the system to file a tax return.

[0073] "Document data" refers to digital image or text data, such as receipts and certificates, that contains information necessary for filing tax returns.

[0074] "Terminal" refers to an electronic device, such as a smartphone, PC, or tablet, that a user uses to acquire document data and send it to a server.

[0075] "Server" refers to the computer system that analyzes uploaded document data and generates and processes the necessary information.

[0076] "Analysis" refers to the process of extracting and classifying important information from uploaded document data.

[0077] "Image processing technology" refers to technology for extracting text information from image data, specifically using technologies such as OCR (optical character recognition).

[0078] "Natural language processing technology" refers to technology for analyzing extracted text data and identifying meaning and important information.

[0079] "JSON format" refers to a lightweight data exchange format that represents data in a structured manner and facilitates data exchange between systems.

[0080] "Electronic filing system" refers to an online system for submitting tax returns via the Internet.

[0081] "Interface" refers to the screen or operating means by which a user interacts with a system and inputs or modifies required information.

[0082] A "secure communication protocol" refers to a communication technology for encrypting data and transmitting it securely, specifically using SSL / TLS.

[0083] "Internal verification" refers to the process of verifying the accuracy of generated returns.

[0084] This invention is a system for easily filing tax returns, which automatically and efficiently performs a series of processes, such as allowing users to acquire document data, uploading it to a server, analyzing it, generating it, confirming it, and submitting it. Specific embodiments of this system are described below.

[0085] Acquiring document data

[0086] Users can take pictures of receipts and certificates using devices such as smartphones or PCs, and the images are saved in the device's local storage.

[0087] For example, when a user takes a photo of a hometown tax donation receipt with their smartphone camera, the image file is saved in the smartphone's "Pictures" folder.

[0088] Uploading data

[0089] The user launches the Tax Return Assistant app and selects the saved image file. The device then uses SSL / TLS encrypted communication to upload the image file to the server.

[0090] Specifically, when a user presses the "Upload" button within the app, the selected image is sent to the server via secure communication, using a secure communication protocol such as SSL / TLS.

[0091] Data analysis

[0092] The server uses image processing and natural language processing technologies to analyze the received image files. First, it uses an OCR (Optical Character Recognition) library as image processing technology to extract text information from the image. Next, it analyzes the extracted text data using a natural language processing library to identify important information such as the amount and the name of the local government.

[0093] Specifically, it uses Tesseract to extract text from images and spaCy, an advanced natural language processing library, to analyze the data.

[0094] Generate and verify information

[0095] The server generates the information necessary for filing tax returns based on the analysis results and sends that information in JSON format to the terminal. An interface is provided for the user to check the information displayed on the terminal screen and make corrections or additions as necessary.

[0096] For example, the information sent from the server can be displayed in a form within the app, and the "amount" field can be manually corrected.

[0097] Automatic generation and submission of tax returns

[0098] The server automatically generates tax returns based on the confirmed data. The generated tax returns undergo internal verification, and if there are no problems, they are submitted to the electronic tax return system. The entire process is automated, allowing users to easily file accurate tax returns.

[0099] For example, by linking to the e-Tax system, tax return data can be sent in CSV format, and the process of officially submitting the tax return can proceed.

[0100] Prompt Sentence Examples

[0101] Below are some example prompts to explain the details of this system to the generative AI model:

[0102] Prompt: Please explain the detailed process of the assistant system for easily filing tax returns. Please explain the specific steps involved in obtaining document data, uploading data, analyzing data, generating and verifying information, and automatically generating and submitting tax returns.

[0103] As a result, by using the system of the present invention, users can significantly reduce time and effort and efficiently file tax returns.

[0104] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0105] Step 1:

[0106] A user takes a picture of a receipt or certificate using the camera on their smartphone or computer. The input is the receipt or certificate, and the output is an image file. The captured image is saved in the device's local storage. Specifically, the user takes the picture using a camera app, and the image file is saved in, for example, a "Pictures" folder.

[0107] Step 2:

[0108] The user launches the Tax Return Assistant app and selects the saved image file. The input is the image file generated in step 1, and the output is the selected image file. Next, the device uploads the image file to the server using SSL / TLS. Specifically, the user taps the "Upload" button in the app, and the device sends the image to the server via encrypted communication.

[0109] Step 3:

[0110] The server analyzes the received image file. The input is the image file sent from the terminal, and the output is text data. First, the server uses the Tesseract library as an image processing technology to extract text information from the image. Specifically, the OCR process scans the image and converts the text information into text data.

[0111] Step 4:

[0112] The server analyzes the text data using natural language processing technology. The input is the text data extracted in step 3, and the output is structured data (e.g., amount, name of local government, date, etc.). Specifically, the server uses the spaCy library to analyze the text data and identify important information such as amount and name of local government.

[0113] Step 5:

[0114] The server generates the information required for tax returns in JSON format based on the analysis results and sends this JSON data to the terminal. The input is the structured data analyzed in step 4, and the output is JSON format data. Specifically, the server appropriately formats the analysis results and sends the data to the user's terminal.

[0115] Step 6:

[0116] The terminal displays the received JSON data on the application screen. The input is the JSON data sent from the server, and the output is an interface that allows the user to check and edit the data. The user uses this interface to check the data and manually edit or add data as needed. Specifically, the data is displayed on a form screen within the app, and the user operates it using the "Confirm" and "Edit" buttons.

[0117] Step 7:

[0118] The server automatically generates tax returns based on the confirmed and corrected data. The input is the data confirmed and corrected by the user, and the output is the tax return. Specifically, the server runs a script that creates a formatted tax return based on this data, and generates the return in a format such as PDF.

[0119] Step 8:

[0120] The server internally verifies the generated tax return and, if there are no problems, submits it to the electronic filing system. The input is the tax return generated in step 7, and the output is a notification of submission completion to the electronic filing system. Specifically, the server verifies the contents of the generated tax return and sends the data to an electronic filing system such as e-Tax for official submission.

[0121] These steps allow users to file their tax returns easily and accurately.

[0122] (Application example 1)

[0123] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0124] Modern individuals and business owners must manage numerous documents, such as receipts and certificates, on a daily basis, and they must accurately organize these documents, especially when filing tax returns. However, manually organizing these documents, extracting the necessary information, and preparing tax returns is a time-consuming and labor-intensive task. There is also a need for technology to efficiently record and manage purchase history when making purchases in stores. Therefore, there is a need for a system that automates these cumbersome and time-consuming tasks and allows users to easily and accurately file tax returns.

[0125] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0126] In this invention, the server includes: [means for a user to read document data such as receipts and certificates;] [means for uploading the read data to the server;] [means for analyzing the data on the server and generating information necessary for filing a tax return;] [means for sending the generated information to the user's terminal so the user can confirm and correct it;] [means for automatically generating a tax return based on the confirmed information and submitting it to an electronic filing system; and [means for using an application installed on a smartphone, smart glasses, or head-mounted display to easily manage the read data at the store via electronic device.] This allows users to easily manage receipt and document data, efficiently organize the information necessary for filing a tax return, and quickly and accurately generate and submit a tax return.

[0127] A "receipt" is a document issued to prove the amount paid in a transaction.

[0128] A "certificate" is an official document issued to certify a particular fact.

[0129] "Document data" refers to document information captured in digital form using a device such as a scanner or camera.

[0130] A "server" is a computer system that provides data and services over a network.

[0131] "Analysis" is the process of breaking down data and understanding and extracting its structure and meaning.

[0132] "Filing a tax return" is the process by which an individual or corporation reports the amount of tax due to tax authorities based on income and expenses for a certain period of time.

[0133] "Information generation" is the process of creating new information based on acquired and analyzed data.

[0134] "Transmission" is the process of moving data from one place to another.

[0135] "Confirmation" refers to the act of the user checking whether the generated information is correct.

[0136] "Correction" refers to the act of correcting any errors or missing information discovered as a result of verification.

[0137] A "tax return" is an official document that contains the information required to file a tax return.

[0138] "Automatic generation" is the process by which a system automatically creates documents and data without the need for manual input.

[0139] The "electronic filing system" is an online system for submitting tax returns to tax authorities via the Internet.

[0140] "Electronic devices" are digital devices such as smartphones, smart glasses, and head-mounted displays.

[0141] A "smartphone" is a multi-functional mobile device that can access the Internet and run applications in addition to the functions of a mobile phone.

[0142] "Smart glasses" are glasses-type mobile devices that can display information in front of the eyes.

[0143] A "head-mounted display" is a device that a user wears on their head to display visual information.

[0144] An "application" is a software program that runs on a computer to provide a particular function or service.

[0145] The system of this invention consists of three main elements: a user, a terminal, and a server. This system allows users to easily manage document data such as receipts and certificates, and efficiently generate and submit the information required for tax returns. The details of this system are described below.

[0146] Overall system configuration

[0147] 1. Users

[0148] Users acquire document data using electronic devices such as smartphones, smart glasses, and head-mounted displays (HMDs). For example, when a user wants to manage products purchased at a store, they acquire the data by taking a photo of the receipt with their smartphone. This document data is then temporarily stored in the local storage of the electronic device.

[0149] 2. Terminal

[0150] The terminal is responsible for uploading the document data acquired by the user to the server. The selected images are sent to the server while maintaining appropriate security through a dedicated tax return assistant application. This application is installed on a smartphone, smart glasses, or HMD.

[0151] 3. Server

[0152] The server analyzes the received document data and generates the information necessary for filing tax returns. Image recognition technology is used to extract text from the document data, and natural language processing technology is used to identify important information such as the amount and store name. The analysis results are sent to the terminal as structured data in JSON format or similar.

[0153] Program processing explanation

[0154] First, the user takes an image of the receipt using the smartphone camera. This data is saved in the smartphone's local storage. Next, the user launches the dedicated tax return assistant app and uploads the saved image data to the server. The uploaded data is analyzed by the server, and text information is extracted from the image.

[0155] The server analyzes the extracted text data using natural language processing technology to identify important information such as the amount and store name. The identified information is generated as structured data in JSON format and sent to the device. The user can then review the generated information on their device and manually correct or add additional information as necessary.

[0156] Finally, the server automatically generates a tax return based on the verified data and submits it to the electronic tax return system. Through this series of processes, users can easily and accurately file their tax returns.

[0157] Specific examples

[0158] For example, suppose a user goes shopping at a store and takes a photo of the receipt with their smartphone. This image is uploaded to the server via the app, and the server analyzes it. The analysis results include the amount and store name, and this information is sent to the device in JSON format. The user checks this information and makes any corrections, after which a final tax return is automatically generated. This tax return is then submitted to the electronic filing system.

[0159] Prompt Sentence Examples

[0160] "Please take a photo of your store receipts, organize the information, and automatically generate and provide your tax return."

[0161] This system allows users to automate cumbersome procedures and save a great deal of time and effort, making it a particularly useful tool for business people and freelancers to improve the efficiency of their daily work.

[0162] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0163] Step 1:

[0164] A user takes an image of a receipt or certificate using an electronic device such as a smartphone, using a dedicated application.

[0165] Input: Physical documents such as receipts and certificates and smartphone

[0166] Data processing: Use the smartphone's camera to capture digital images of documents.

[0167] Output: Image file of document data (e.g., JPEG format)

[0168] Step 2:

[0169] The device (smartphone) temporarily stores the captured image in local storage.

[0170] Input: Image file of photographed document data

[0171] Data processing: Image files are saved to the smartphone's local storage.

[0172] Output: Image file saved in local storage

[0173] Step 3:

[0174] The device (smartphone) uploads image data to the server via a dedicated app, with appropriate security measures taken during uploading.

[0175] Input: Image file stored in local storage

[0176] Data processing: Send the image file to the server using an HTTP POST request.

[0177] Output: Image file uploaded to the server

[0178] Step 4:

[0179] The server analyzes the received image data. First, it uses image recognition technology to extract text information from the image.

[0180] Input: Image file uploaded to the server

[0181] Data processing: Extract text from images using OCR (Optical Character Recognition) technology.

[0182] Output: Extracted text data

[0183] Step 5:

[0184] The server analyzes the extracted text data using natural language processing technology to identify important information such as the amount and store name.

[0185] Input: Extracted text data

[0186] Data processing: Analyze text data using natural language processing techniques and extract important information (e.g., amount, store name).

[0187] Output: Structured data in JSON format (including important information)

[0188] Step 6:

[0189] The server sends the generated JSON format data to the terminal, where the user can check and modify it.

[0190] Input: Structured data in JSON format

[0191] Data processing: Data is sent to the device and displayed in a dedicated application. The user can check and edit the information.

[0192] Output: Data reviewed and corrected by the user

[0193] Step 7:

[0194] The server automatically generates a tax return based on the confirmed and corrected data and validates it internally.

[0195] Input: Data modified by the user

[0196] Data processing: Using generative AI models, tax returns are automatically generated and the data consistency is verified internally.

[0197] Output: Auto-generated tax return

[0198] Step 8:

[0199] The server submits the automatically generated tax return to the electronic filing system.

[0200] Input: Auto-generated tax return

[0201] Data processing: Send data based on the submission protocol to the electronic reporting system.

[0202] Output: Tax return submitted to the electronic filing system

[0203] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0204] The tax return assistant system of this invention combines an emotion engine to improve the user experience, allowing individual users to easily file tax returns. The system mainly works as follows: the user reads document data, uploads it to a server, the server analyzes and generates the data, and then the emotion engine recognizes the user's emotional state and allows the user to confirm and correct the data before automatically generating the tax return, which is finally submitted to the electronic filing system.

[0205] Technical configuration

[0206] 1. Acquisition of document data

[0207] The user uses the device to take an image of a receipt or certificate, which then captures the document data and temporarily stores it in the device's local storage.

[0208] 2. Uploading data

[0209] The user uploads the acquired document data to the server through a dedicated tax return assistant app. The device then sends the user-selected images to the server while maintaining appropriate security.

[0210] 3. Data analysis (image recognition)

[0211] The server receives the uploaded image data and extracts text from the document image using image processing technology such as OpenCV. A character recognition engine such as PyTesseract then extracts the text from the image as text data.

[0212] 4. Data Analysis (Natural Language Processing)

[0213] The server uses natural language processing technology to analyze the text data it acquires and extracts the information necessary for filing tax returns (e.g., amount, date, name of local government). During this process, the AI ​​algorithm identifies specific patterns and keywords, and categorizes and organizes the information.

[0214] 5. Emotion Recognition by Emotion Engine

[0215] An emotion engine installed on the server or device analyzes the user's facial image and voice data to recognize their emotional state. The emotional data acquired by the emotion engine is fed back to the entire system and used to improve the user experience. For example, if the user is under high stress, the interface design or messages can be changed.

[0216] 6. Information Generation and Verification

[0217] Based on the analyzed data and feedback from the emotion engine, the server generates data summarizing the information required for filing tax returns. This generated data is sent to the terminal and displayed to the user. The user checks the displayed information and manually corrects or adds anything necessary. The emotion engine monitors the user's emotional state throughout this process, displaying help information and advice as appropriate.

[0218] 7. Automatic generation of tax returns

[0219] The server automatically generates a tax return based on the final confirmed information, and the generated return is verified internally to ensure its accuracy.

[0220] 8. Submission to the electronic filing system

[0221] The server submits the generated tax return to the official electronic tax return system. Once the submission is complete, the server sends a notification to the user that the return has been completed. This notification can be sent via email or in-app notification.

[0222] Specific examples

[0223] For example, when a user wants to use a receipt for hometown tax donations, they take a photo of the receipt with their smartphone camera and upload the image to the server via the app. The server analyzes the image, extracts the amount and name of the local government, and then uses an emotion engine to recognize the user's emotional state. Based on the extracted data and emotion data, the most appropriate information and advice is displayed on the user's device. After the user confirms and corrects it, the server finally generates a tax return and submits it to the electronic filing system, completing the tax return.

[0224] In this way, the system of the present invention simplifies the complicated procedures for users and provides optimal support according to each individual's emotional state, thereby saving a great deal of time and effort.

[0225] The processing flow will be explained below.

[0226] Step 1: Scan your documents

[0227] The user uses the smartphone camera to take a photo of a document such as a receipt or certificate. The image is saved in the device's local storage. The user then launches the dedicated tax return assistant app and selects the image they took.

[0228] Step 2: Upload your data

[0229] The user uploads the saved document data to the server through the app. The device sends the selected image to the server using a secure communication protocol (such as SSL). The upload progress is displayed within the app.

[0230] Step 3: Data analysis (image recognition)

[0231] The server receives the uploaded image data. Next, it uses image processing technology such as OpenCV to extract characters from the image. Then it uses a character recognition engine such as PyTesseract to obtain the text in the image as text data.

[0232] Step 4: Data analysis (natural language processing)

[0233] The server analyzes the acquired text data using natural language processing technology. Specifically, it extracts important information from the text, such as amounts, dates, and names of local governments. During this process, an AI algorithm identifies specific patterns and keywords and organizes the information.

[0234] Step 5: Emotion Recognition with the Emotion Engine

[0235] The device's camera and microphone are used to capture the user's facial image and voice data, and an emotion engine installed on the device or server analyzes this data to identify the user's emotional state.

[0236] Step 6: Generate feedback

[0237] Based on the emotional data acquired by the emotion engine, the server generates feedback and advice to improve the user experience. For example, if the user is feeling stressed, a soothing message or problem-solving advice will be displayed on the device.

[0238] Step 7: Verify and correct information

[0239] The server generates the information necessary for filing tax returns based on the analysis results. This information is sent to the device in JSON format or similar and displayed to the user. The user checks the displayed information and makes corrections or adds additional information as necessary. The emotion engine also operates during this process, providing help and guidance as appropriate.

[0240] Step 8: Auto-generate tax returns

[0241] The server automatically generates a tax return based on the verified information, which is then internally verified to ensure accuracy.

[0242] Step 9: Submit to the electronic filing system

[0243] The server submits the final generated tax return to the official electronic tax return system. Once the submission is complete, the server sends a notification to the user that the return has been completed. This notification can be sent via email or in-app notification.

[0244] These steps allow users to file their tax returns accurately and quickly with minimal effort, and the emotional feedback engine improves the user experience, making the filing process more convenient.

[0245] Example 2

[0246] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0247] Traditional tax return systems require manual data entry and complex procedures, placing a heavy burden on users. Furthermore, due to the lack of emotional state recognition and feedback, tax return filing is often done under stress, increasing the likelihood of errors and delays in entry.

[0248] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0249] In this invention, the server includes: [means for analyzing data on the server and generating information necessary for filing tax returns;] [means for sending the generated information to the user's terminal so that the user can confirm and correct it; and] [means for acquiring the user's emotional state and feeding that data back to the entire system.] This makes it possible [to simplify the complicated tax return procedure and provide optimal support according to the user's emotional state].

[0250] "User" refers to an individual or corporation that uses the System to file tax returns.

[0251] "Document data" refers to data that has been photographed and stored digitally from physical documents such as receipts and certificates.

[0252] "Terminal" refers to a device such as a smartphone, tablet, or PC that is used to photograph, save, and upload document data to a server.

[0253] "Server" refers to the computer system that analyzes uploaded document data and generates and manages the information necessary for filing tax returns.

[0254] "Emotion engine" refers to artificial intelligence technology that analyzes a user's facial images and voice data to recognize their emotional state.

[0255] "Image processing technology" refers to computer vision technology used to extract character strings from document data (images).

[0256] "Natural language processing technology" refers to the technology used to analyze acquired text data and extract and classify information necessary for filing tax returns.

[0257] "Interface" refers to the user interface through which a user interacts with the system and makes any necessary corrections or additional inputs.

[0258] "Electronic filing system" means a system provided by a government or related authority for filing tax returns electronically.

[0259] "Automatic generation" refers to the process by which a server automatically generates tax returns based on specific rules and algorithms.

[0260] "Feedback" refers to the process by which the emotion engine makes adaptive adjustments to the entire system based on the user's emotional state.

[0261] MODE FOR CARRYING OUT THE INVENTION

[0262] The tax return assistant system of this invention is designed to allow individuals and corporations to easily file tax returns, and aims to improve the user experience by combining it with an emotion engine. The system mainly operates in a series of steps: the user reads document data, uploads it to a server, the server analyzes and generates the data, and then automatically generates a tax return after recognizing the user's emotional state using the emotion engine, and finally submits it to an electronic filing system.

[0263] Hardware and software used

[0264] Device: The user's smartphone, tablet, computer, etc., from which document data is acquired and uploaded to the server.

[0265] Server: Analyzes and generates data, recognizes emotions using an emotion engine, automatically generates tax returns, and submits them to the electronic tax return system.

[0266] Image processing technology: OpenCV is used to perform image recognition of document data.

[0267] Character recognition engine: Uses PyTesseract to extract characters in an image as text data.

[0268] Natural Language Processing (NLP): Uses technology to extract and organize information required for tax returns.

[0269] Emotion Engine: An artificial intelligence technology that analyzes a user's facial images and voice data to recognize their emotional state.

[0270] Specific example explanation

[0271] For example, consider the case where a user uses a receipt for hometown tax donations. The user takes a photo of the receipt using the camera on their smartphone. This document data is then uploaded to the server using the tax return assistant app. The server preprocesses the image using image processing technology (OpenCV) and extracts the text from the image using PyTesseract. It then uses natural language processing technology to identify necessary information such as the "amount" and "name of the local government" from the extracted text.

[0272] At the same time, the emotion engine analyzes the user's facial image and voice data to assess their emotional state (e.g., stress level). Based on this emotion data, the server provides the user with an optimal interface and displays necessary advice and help information.

[0273] Based on the analyzed data and emotional feedback, the information necessary for filing tax returns is generated and sent to the user's device. The user can then review the displayed information and make corrections or additional entries as necessary. During this process, the emotion engine monitors the user's emotional state and responds accordingly.

[0274] The server then automatically generates a tax return based on the final confirmed information and submits it to the electronic filing system. Once submission is complete, the server sends a notification of completion to the user via email or in-app notification.

[0275] Example prompts for generative AI models

[0276] "Please extract the amount and name of the local government from the hometown tax receipt, use an emotion engine to determine the user's stress level, and then automatically generate a tax return."

[0277] As described above, this system simplifies the complicated procedures users go through and provides optimal support according to their individual emotional state, thereby saving them a great deal of time and effort.

[0278] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0279] Program processing flow

[0280] Step 1:

[0281] The user takes a photo of the receipt or certificate required for tax return filing using a device such as a smartphone or PC. The captured image is temporarily saved in the device's local storage.

[0282] Specific behavior:

[0283] Input: receipt or certificate

[0284] Processing: Take a photo using the device's camera

[0285] Output: Image data saved in local storage

[0286] Step 2:

[0287] The user opens the Tax Return Assistant app and instructs it to upload the saved document data to the server. The device encrypts the image selected by the user and sends it to the server via a secure communication channel.

[0288] Specific behavior:

[0289] Input: Image data stored in local storage

[0290] Processing: Encryption of image data, transmission using HTTPS

[0291] Output: Image data uploaded to the server

[0292] Step 3:

[0293] The server analyzes the received image data using technologies such as OpenCV and PyTesseract, extracts character strings from the image, and saves them as text data.

[0294] Specific behavior:

[0295] Input: Image data uploaded to the server

[0296] Processing: Image preprocessing with OpenCV, character recognition with PyTesseract

[0297] Output: Text data

[0298] Step 4:

[0299] The server analyzes the acquired text data using natural language processing technology. The server identifies specific keywords and patterns, and extracts and organizes the necessary information (e.g., amount, date, name of local government).

[0300] Specific behavior:

[0301] Input: Text data

[0302] Processing: Applying NLP models to extract information

[0303] Output: Extracted information data (e.g. amount, date, name of local government)

[0304] Step 5:

[0305] The server or device uses an emotion engine to analyze the user's facial image and voice data to recognize the user's emotional state. The recognized emotional data is used throughout the system.

[0306] Specific behavior:

[0307] Input: User's facial image and voice data

[0308] Processing: Emotion recognition by emotion engine

[0309] Output: Emotion data

[0310] Step 6:

[0311] Based on the analyzed data and emotional feedback, the server generates optimal tax return information. The generated information is sent to the device, prompting the user to confirm and correct it. The emotion engine monitors the user's status throughout this process, providing help and advice as needed.

[0312] Specific behavior:

[0313] Input: Extracted information data, emotion data

[0314] Processing: Data formatting and generation

[0315] Output: Generated tax return information

[0316] Step 7:

[0317] The server automatically generates an official tax return based on the final verified information, which is then internally verified to ensure accuracy.

[0318] Specific behavior:

[0319] Input: Tax return information

[0320] Processing: Automatic generation and internal validation of tax returns

[0321] Output: Generated tax return

[0322] Step 8:

[0323] The server submits the verified tax return to the official electronic tax return system. Once the submission is complete, the server sends a notification to the user that the return has been completed. This notification can be sent via email or in-app notification.

[0324] Specific behavior:

[0325] Input: Generated tax return

[0326] Processing: Submission using the API of the electronic reporting system

[0327] Output: Notification of declaration completion

[0328] _Example of prompt sentence for generative AI model_

[0329] "Please extract the amount and name of the local government from the hometown tax receipt, use an emotion engine to determine the user's stress level, and then automatically generate a tax return."

[0330] (Application example 2)

[0331] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0332] In conventional tax return systems, users had to manually collect and analyze document data and generate tax returns, which was a cumbersome process that required time and effort. Furthermore, there was no mechanism to reduce the emotional burden caused by the stress and hassle, resulting in a poor user experience. Furthermore, there was no mechanism to generate and deliver interactive content based on the user's emotional state, making it difficult to personalize content based on the emotional state of each individual user.

[0333] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: [means for a user to read document data such as receipts and certificates;] [means for uploading the read data to the server;] [means for analyzing the data at the server and generating information necessary for filing a tax return;] [means for sending the generated information to the user's terminal so that the user can confirm and correct it;] [means for automatically generating a tax return based on the confirmed information and submitting it to an electronic filing system;] [means for optimizing the interface to the user based on the user's emotional state; and [means for generating story content based on the user's emotional data and delivering it interactively.] This simplifies the tax return process, reduces the emotional burden on the user, and enables personalized interactive content delivery based on the user's emotional state.

[0334] "User" refers to an individual who uses an application or system.

[0335] "Document data such as receipts and certificates" refers to digital data of official documents intended for the transfer of money or official certification.

[0336] "Server" refers to a computer system that stores, processes, and distributes data over a network.

[0337] "Analyzing data" refers to the process of processing acquired digital data and converting it into useful information.

[0338] "Information required for filing tax returns" refers to data such as amounts, dates, and personal information that are required when filing a tax return.

[0339] "Terminal" refers to a computer device such as a mobile phone, tablet, or PC that a user uses.

[0340] "Verify and correct" refers to the operation in which the user reviews the generated information and makes changes as necessary.

[0341] "Automatically generating tax returns" refers to the process by which the system automatically generates the necessary tax return documents based on the collected and analyzed data.

[0342] "Electronic filing system" refers to the official online platform for filing tax returns via the Internet.

[0343] "Emotional state" refers to data that indicates the user's mood or emotions at that time.

[0344] "Optimizing the interface" refers to improving the user experience by changing the system's display method and operating procedures according to the user's emotional state.

[0345] "Story content" refers to digital content in the form of a story that users can enjoy.

[0346] "Delivered interactively" means providing content in a format that allows users to progress through the story through choices.

[0347] The following description is given as a mode for carrying out the invention.

[0348] The present invention includes a tax return assistant system and an interactive story content distribution system. The system aims to enable individual users to complete tax return procedures easily and efficiently, and to provide personalized content based on the user's emotional state.

[0349] Hardware and Software

[0350] 1. Hardware

[0351] Devices: Smartphones, tablets, PCs, head-mounted displays, etc.

[0352] Server: A server capable of high-performance data processing

[0353] 2. Software

[0354] Image processing technology: OpenCV, PyTesseract

[0355] Natural language processing technology: NLTK, Transformers library

[0356] Emotion Recognition Engine: Emotion Recognition Model Using TensorFlow

[0357] Cloud storage: AWS S3

[0358] Web services: AWS Lambda, API Gateway

[0359] Details of data processing and calculation

[0360] 1. Obtaining and uploading document data

[0361] A user uses a device such as a smartphone to read document data such as receipts or certificates. The acquired data is temporarily stored in the device's local storage and then uploaded to a server via a dedicated application. This process uses the SSL / TLS protocol to maintain data confidentiality.

[0362] 2. Image Recognition and Text Data Extraction

[0363] The server uses OpenCV and PyTesseract to analyze uploaded document images and extract text data, such as the amount, date, and store name written on the receipt.

[0364] 3. Natural Language Processing of Text Data

[0365] The extracted text data is analyzed using NLTK and Transformers libraries. The server then accurately extracts and classifies the information required for tax returns. For example, it automatically assigns corresponding data to fields such as "amount," "date," and "address."

[0366] 4. Emotion Recognition and User Interface Optimization

[0367] An emotion recognition engine installed on the device or server analyzes the user's facial image and voice data to recognize their emotional state. This data is fed back into the interface design and help information display, helping to reduce the user's stress.

[0368] 5. Automatic generation and submission of tax return documents

[0369] The server automatically generates a tax return based on the data checked and corrected by the user. The generated return undergoes an internal validation process and is then submitted to the electronic filing system. After submission is complete, the server sends a completion notification to the user.

[0370] 6. Interactive story content generation and distribution

[0371] The server uses a generative AI model to generate story content based on the user's emotional state. Emotional data is used as prompts to dynamically generate story scenes. This content is delivered in an interactive format, allowing the user to progress through the story through choices.

[0372] Specific examples

[0373] For example, if a user is uploading image data for tax return and the process is stressful, the emotion recognition engine will detect this and provide a relaxing interface and friendly guidance for specific steps. After completing the tax return, the system will provide personalized story content to refresh the user.

[0374] Prompt Sentence Examples

[0375] Generate the following story scene based on the user's emotional data. The emotional state is excitement. The characters are a brave knight and a dragon. The location is a dark cave. The tone of the story is exciting and tense.

[0376] In this way, the system of the present invention can improve the user experience by simplifying the user's procedures and providing optimal support and entertainment tailored to their emotional state.

[0377] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0378] Step 1:

[0379] The user obtains document data such as receipts and certificates.

[0380] Specifically, the user takes an image of a document using a device such as a smartphone and temporarily stores it in the device's local storage.

[0381] Input: Document data (image format)

[0382] Output: Document data saved in local storage

[0383] Step 2:

[0384] The terminal uploads the document data to the server.

[0385] Specifically, the system uses a dedicated tax return assistant app to send the image selected by the user from the device to the server, while maintaining data confidentiality using the SSL / TLS protocol.

[0386] Input: Document data stored in local storage

[0387] Output: Document data uploaded to the server

[0388] Step 3:

[0389] The server analyzes the document data and performs image recognition.

[0390] Specifically, the server analyzes document images using OpenCV and PyTesseract and extracts text data.

[0391] Input: Document data uploaded to the server

[0392] Output: Extracted text data

[0393] Step 4:

[0394] The server analyzes the extracted text data using natural language processing.

[0395] Specifically, the server uses NLTK and Transformers libraries to analyze text data, extracting and classifying information necessary for tax returns. For example, it automatically identifies data corresponding to fields such as "amount," "date," and "address."

[0396] Input: Extracted text data

[0397] Output: Analyzed information (data required for tax return)

[0398] Step 5:

[0399] The server transmits the analyzed information to the user terminal.

[0400] Specifically, the information is sent to the user's device appropriately so that the user can check and modify it, and the interface is optimized to reduce user stress.

[0401] Input: Parsed information

[0402] Output: Information sent to the user's device

[0403] Step 6:

[0404] The terminal recognizes the user's emotional state.

[0405] Specifically, the system collects facial images and voice data using the device's camera and microphone, and determines the user's emotional state using an emotion recognition engine using TensorFlow.

[0406] Input: face image, voice data

[0407] Output: Emotional state data

[0408] Step 7:

[0409] A server generates story content based on emotional states.

[0410] Specifically, it uses emotion data as prompts to generate digital content (e.g., story scenes) using generative AI models, specifically generating exciting and suspenseful stories based on heightened emotional states.

[0411] Input: Emotional state data

[0412] Output: Generated story content

[0413] Step 8:

[0414] The server interactively delivers the generated story content to the user.

[0415] Specifically, the generated content is delivered to the user's device, allowing the user to progress through the story through choices. This delivery is done in real time, and the story changes dynamically depending on the user's choices.

[0416] Input: Generated story content

[0417] Output: Interactive story content delivered to user devices

[0418] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0419] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0420] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0421] [Second embodiment]

[0422] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0423] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0424] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0425] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0426] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0427] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0428] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0429] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0430] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0431] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0432] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0433] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0434] The tax return assistant system of this invention is designed to allow individual users to easily file tax returns. The system mainly works by having the user read document data, upload it to a server, analyze and generate the data, have the user check and correct it, automatically generate a tax return, and finally submit it to the electronic filing system.

[0435] Technical configuration

[0436] 1. Acquisition of document data

[0437] The user takes a photo of the receipt or certificate using a device such as a smartphone or computer. This captures the document data, which is then temporarily stored in the local storage of the user's device.

[0438] 2. Uploading data

[0439] The user uploads the acquired document data to the server through a dedicated tax return assistant app. The device then sends the user-selected images to the server while maintaining appropriate security.

[0440] 3. Data Analysis

[0441] The server analyzes the received document data using image processing and natural language processing technologies. First, it performs image recognition and extracts the text information written on the document as text. This extracted text data is then analyzed using natural language processing technology to identify important information such as the amount and name of the local government.

[0442] 4. Information Generation and Verification

[0443] Based on the parsed data, the server generates the information necessary for filing tax returns. This information is sent to the terminal as structured data in JSON format or similar and displayed to the user. The user can then check the displayed information and manually correct or add anything necessary.

[0444] 5. Automatic generation and submission of tax returns

[0445] The server automatically generates a tax return based on the final confirmed data. The generated tax return is then verified internally and then submitted to the electronic filing system. This series of processes allows users to easily file accurate tax returns.

[0446] Specific examples

[0447] For example, let's say a user uses a receipt for hometown tax donations. First, they take a photo of the receipt with their smartphone camera and upload the image to the server via the app. The server analyzes the image and extracts the amount and name of the local government. The extracted data is sent to the user's device, where the user can confirm and correct it. Finally, the server generates a tax return based on this information and submits it to the electronic filing system. This completes the tax return.

[0448] In this way, the system of the present invention allows users to simplify cumbersome procedures and save time and effort.

[0449] The processing flow will be explained below.

[0450] Step 1: Scan your documents

[0451] The user uses the camera function to take a photo of the document data and save it on the device. The user then launches a dedicated app and selects the image they took.

[0452] Step 2: Upload your data

[0453] The device sends the document data to the server. During transmission, the data is transferred securely using security protocols such as SSL. The upload progress is displayed on the user's device.

[0454] Step 3: Data analysis (image recognition)

[0455] The server receives the uploaded image data. Next, it uses image processing technology such as OpenCV to extract text from the document image. Then it uses a character recognition engine such as PyTesseract to recognize the text in the image and obtain the text data.

[0456] Step 4: Data analysis (natural language processing)

[0457] The server uses natural language processing technology to analyze the text data it acquires and extracts the information necessary for filing tax returns (e.g., amount, date, name of local government). During this process, the AI ​​algorithm identifies specific patterns and keywords, and categorizes and organizes the information.

[0458] Step 5: Generate and send information

[0459] Based on the analysis results, the server generates data summarizing the information required for filing, and this data is sent to the device in a structured format such as JSON.

[0460] Step 6: Verify and correct information

[0461] The device displays the received data to the user. The user can check the displayed information and manually edit or add anything if necessary. Once edits are complete, the user taps the "Confirm" button.

[0462] Step 7: Auto-generate tax returns

[0463] The server automatically generates tax returns based on the verified information, and the generated returns are validated internally to ensure accuracy.

[0464] Step 8: Submit to the electronic filing system

[0465] The server submits the generated tax return to the official electronic tax return system. Once the submission is complete, the server sends a notification to the user that the return has been completed. The notification can be sent via email or in-app notification.

[0466] These steps allow users to accurately file their tax returns with minimal effort on their part.

[0467] Example 1

[0468] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0469] Accounting and tax return procedures are extremely complicated and require a great deal of time and effort, especially for individual users. As a result, many users struggle to manage the necessary documents and prepare accurate tax returns. Furthermore, manual data entry and correction increases the risk of human error. Therefore, there is a need for a system that simplifies tax return procedures and processes them accurately and quickly.

[0470] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0471] In this invention, the server includes: [means using image processing technology to perform image recognition of document data]; [means using natural language processing technology to analyze text data extracted from the image]; and [means using JSON format to structuring data]. This enables users to easily digitize document data such as receipts and certificates, convert that data into information necessary for tax returns, automatically generate tax returns accurately and quickly, and submit them to the electronic filing system.

[0472] "User" refers to a person who uses the system to file a tax return.

[0473] "Document data" refers to digital image or text data, such as receipts and certificates, that contains information necessary for filing tax returns.

[0474] "Terminal" refers to an electronic device, such as a smartphone, PC, or tablet, that a user uses to acquire document data and send it to a server.

[0475] "Server" refers to the computer system that analyzes uploaded document data and generates and processes the necessary information.

[0476] "Analysis" refers to the process of extracting and classifying important information from uploaded document data.

[0477] "Image processing technology" refers to technology for extracting text information from image data, specifically using technologies such as OCR (optical character recognition).

[0478] "Natural language processing technology" refers to technology for analyzing extracted text data and identifying meaning and important information.

[0479] "JSON format" refers to a lightweight data exchange format that represents data in a structured manner and facilitates data exchange between systems.

[0480] "Electronic filing system" refers to an online system for submitting tax returns via the Internet.

[0481] "Interface" refers to the screen or operating means by which a user interacts with a system and inputs or modifies required information.

[0482] A "secure communication protocol" refers to a communication technology for encrypting data and transmitting it securely, specifically using SSL / TLS.

[0483] "Internal verification" refers to the process of verifying the accuracy of generated returns.

[0484] This invention is a system for easily filing tax returns, which automatically and efficiently performs a series of processes, such as allowing users to acquire document data, uploading it to a server, analyzing it, generating it, confirming it, and submitting it. Specific embodiments of this system are described below.

[0485] Acquiring document data

[0486] Users can take pictures of receipts and certificates using devices such as smartphones or PCs, and the images are saved in the device's local storage.

[0487] For example, when a user takes a photo of a hometown tax donation receipt with their smartphone camera, the image file is saved in the smartphone's "Pictures" folder.

[0488] Uploading data

[0489] The user launches the Tax Return Assistant app and selects the saved image file. The device then uses SSL / TLS encrypted communication to upload the image file to the server.

[0490] Specifically, when a user presses the "Upload" button within the app, the selected image is sent to the server via secure communication, using a secure communication protocol such as SSL / TLS.

[0491] Data analysis

[0492] The server uses image processing and natural language processing technologies to analyze the received image files. First, it uses an OCR (Optical Character Recognition) library as image processing technology to extract text information from the image. Next, it analyzes the extracted text data using a natural language processing library to identify important information such as the amount and the name of the local government.

[0493] Specifically, it uses Tesseract to extract text from images and spaCy, an advanced natural language processing library, to analyze the data.

[0494] Generate and verify information

[0495] The server generates the information necessary for filing tax returns based on the analysis results and sends that information in JSON format to the terminal. An interface is provided for the user to check the information displayed on the terminal screen and make corrections or additions as necessary.

[0496] For example, the information sent from the server can be displayed in a form within the app, and the "amount" field can be manually corrected.

[0497] Automatic generation and submission of tax returns

[0498] The server automatically generates tax returns based on the confirmed data. The generated tax returns undergo internal verification, and if there are no problems, they are submitted to the electronic tax return system. The entire process is automated, allowing users to easily file accurate tax returns.

[0499] For example, by linking to the e-Tax system, tax return data can be sent in CSV format, and the process of officially submitting the tax return can proceed.

[0500] Prompt Sentence Examples

[0501] Below are some example prompts to explain the details of this system to the generative AI model:

[0502] Prompt: Please explain the detailed process of the assistant system for easily filing tax returns. Please explain the specific steps involved in obtaining document data, uploading data, analyzing data, generating and verifying information, and automatically generating and submitting tax returns.

[0503] As a result, by using the system of the present invention, users can significantly reduce time and effort and efficiently file tax returns.

[0504] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0505] Step 1:

[0506] A user takes a picture of a receipt or certificate using the camera on their smartphone or computer. The input is the receipt or certificate, and the output is an image file. The captured image is saved in the device's local storage. Specifically, the user takes the picture using a camera app, and the image file is saved in, for example, a "Pictures" folder.

[0507] Step 2:

[0508] The user launches the Tax Return Assistant app and selects the saved image file. The input is the image file generated in step 1, and the output is the selected image file. Next, the device uploads the image file to the server using SSL / TLS. Specifically, the user taps the "Upload" button in the app, and the device sends the image to the server via encrypted communication.

[0509] Step 3:

[0510] The server analyzes the received image file. The input is the image file sent from the terminal, and the output is text data. First, the server uses the Tesseract library as an image processing technology to extract text information from the image. Specifically, the OCR process scans the image and converts the text information into text data.

[0511] Step 4:

[0512] The server analyzes the text data using natural language processing technology. The input is the text data extracted in step 3, and the output is structured data (e.g., amount, name of local government, date, etc.). Specifically, the server uses the spaCy library to analyze the text data and identify important information such as amount and name of local government.

[0513] Step 5:

[0514] The server generates the information required for tax returns in JSON format based on the analysis results and sends this JSON data to the terminal. The input is the structured data analyzed in step 4, and the output is JSON format data. Specifically, the server appropriately formats the analysis results and sends the data to the user's terminal.

[0515] Step 6:

[0516] The terminal displays the received JSON data on the application screen. The input is the JSON data sent from the server, and the output is an interface that allows the user to check and edit the data. The user uses this interface to check the data and manually edit or add data as needed. Specifically, the data is displayed on a form screen within the app, and the user operates it using the "Confirm" and "Edit" buttons.

[0517] Step 7:

[0518] The server automatically generates tax returns based on the confirmed and corrected data. The input is the data confirmed and corrected by the user, and the output is the tax return. Specifically, the server runs a script that creates a formatted tax return based on this data, and generates the return in a format such as PDF.

[0519] Step 8:

[0520] The server internally verifies the generated tax return and, if there are no problems, submits it to the electronic filing system. The input is the tax return generated in step 7, and the output is a notification of submission completion to the electronic filing system. Specifically, the server verifies the contents of the generated tax return and sends the data to an electronic filing system such as e-Tax for official submission.

[0521] These steps allow users to file their tax returns easily and accurately.

[0522] (Application example 1)

[0523] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0524] Modern individuals and business owners must manage numerous documents, such as receipts and certificates, on a daily basis, and they must accurately organize these documents, especially when filing tax returns. However, manually organizing these documents, extracting the necessary information, and preparing tax returns is a time-consuming and labor-intensive task. There is also a need for technology to efficiently record and manage purchase history when making purchases in stores. Therefore, there is a need for a system that automates these cumbersome and time-consuming tasks and allows users to easily and accurately file tax returns.

[0525] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0526] In this invention, the server includes: [means for a user to read document data such as receipts and certificates;] [means for uploading the read data to the server;] [means for analyzing the data on the server and generating information necessary for filing a tax return;] [means for sending the generated information to the user's terminal so the user can confirm and correct it;] [means for automatically generating a tax return based on the confirmed information and submitting it to an electronic filing system; and [means for using an application installed on a smartphone, smart glasses, or head-mounted display to easily manage the read data at the store via electronic device.] This allows users to easily manage receipt and document data, efficiently organize the information necessary for filing a tax return, and quickly and accurately generate and submit a tax return.

[0527] A "receipt" is a document issued to prove the amount paid in a transaction.

[0528] A "certificate" is an official document issued to certify a particular fact.

[0529] "Document data" refers to document information captured in digital form using a device such as a scanner or camera.

[0530] A "server" is a computer system that provides data and services over a network.

[0531] "Analysis" is the process of breaking down data and understanding and extracting its structure and meaning.

[0532] "Filing a tax return" is the process by which an individual or corporation reports the amount of tax due to tax authorities based on income and expenses for a certain period of time.

[0533] "Information generation" is the process of creating new information based on acquired and analyzed data.

[0534] "Transmission" is the process of moving data from one place to another.

[0535] "Confirmation" refers to the act of the user checking whether the generated information is correct.

[0536] "Correction" refers to the act of correcting any errors or missing information discovered as a result of verification.

[0537] A "tax return" is an official document that contains the information required to file a tax return.

[0538] "Automatic generation" is the process by which a system automatically creates documents and data without the need for manual input.

[0539] The "electronic filing system" is an online system for submitting tax returns to tax authorities via the Internet.

[0540] "Electronic devices" are digital devices such as smartphones, smart glasses, and head-mounted displays.

[0541] A "smartphone" is a multi-functional mobile device that can access the Internet and run applications in addition to the functions of a mobile phone.

[0542] "Smart glasses" are glasses-type mobile devices that can display information in front of the eyes.

[0543] A "head-mounted display" is a device that a user wears on their head to display visual information.

[0544] An "application" is a software program that runs on a computer to provide a particular function or service.

[0545] The system of this invention consists of three main elements: a user, a terminal, and a server. This system allows users to easily manage document data such as receipts and certificates, and efficiently generate and submit the information required for tax returns. The details of this system are described below.

[0546] Overall system configuration

[0547] 1. Users

[0548] Users acquire document data using electronic devices such as smartphones, smart glasses, and head-mounted displays (HMDs). For example, when a user wants to manage products purchased at a store, they acquire the data by taking a photo of the receipt with their smartphone. This document data is then temporarily stored in the local storage of the electronic device.

[0549] 2. Terminal

[0550] The terminal is responsible for uploading the document data acquired by the user to the server. The selected images are sent to the server while maintaining appropriate security through a dedicated tax return assistant application. This application is installed on a smartphone, smart glasses, or HMD.

[0551] 3. Server

[0552] The server analyzes the received document data and generates the information necessary for filing tax returns. Image recognition technology is used to extract text from the document data, and natural language processing technology is used to identify important information such as the amount and store name. The analysis results are sent to the terminal as structured data in JSON format or similar.

[0553] Program processing explanation

[0554] First, the user takes an image of the receipt using the smartphone camera. This data is saved in the smartphone's local storage. Next, the user launches the dedicated tax return assistant app and uploads the saved image data to the server. The uploaded data is analyzed by the server, and text information is extracted from the image.

[0555] The server analyzes the extracted text data using natural language processing technology to identify important information such as the amount and store name. The identified information is generated as structured data in JSON format and sent to the device. The user can then review the generated information on their device and manually correct or add additional information as necessary.

[0556] Finally, the server automatically generates a tax return based on the verified data and submits it to the electronic tax return system. Through this series of processes, users can easily and accurately file their tax returns.

[0557] Specific examples

[0558] For example, suppose a user goes shopping at a store and takes a photo of the receipt with their smartphone. This image is uploaded to the server via the app, and the server analyzes it. The analysis results include the amount and store name, and this information is sent to the device in JSON format. The user checks this information and makes any corrections, after which a final tax return is automatically generated. This tax return is then submitted to the electronic filing system.

[0559] Prompt Sentence Examples

[0560] "Please take a photo of your store receipts, organize the information, and automatically generate and provide your tax return."

[0561] This system allows users to automate cumbersome procedures and save a great deal of time and effort, making it a particularly useful tool for business people and freelancers to improve the efficiency of their daily work.

[0562] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0563] Step 1:

[0564] A user takes an image of a receipt or certificate using an electronic device such as a smartphone, using a dedicated application.

[0565] Input: Physical documents such as receipts and certificates and smartphone

[0566] Data processing: Use the smartphone's camera to capture digital images of documents.

[0567] Output: Image file of document data (e.g., JPEG format)

[0568] Step 2:

[0569] The device (smartphone) temporarily stores the captured image in local storage.

[0570] Input: Image file of photographed document data

[0571] Data processing: Image files are saved to the smartphone's local storage.

[0572] Output: Image file saved in local storage

[0573] Step 3:

[0574] The device (smartphone) uploads image data to the server via a dedicated app, with appropriate security measures taken during uploading.

[0575] Input: Image file stored in local storage

[0576] Data processing: Send the image file to the server using an HTTP POST request.

[0577] Output: Image file uploaded to the server

[0578] Step 4:

[0579] The server analyzes the received image data. First, it uses image recognition technology to extract text information from the image.

[0580] Input: Image file uploaded to the server

[0581] Data processing: Extract text from images using OCR (Optical Character Recognition) technology.

[0582] Output: Extracted text data

[0583] Step 5:

[0584] The server analyzes the extracted text data using natural language processing technology to identify important information such as the amount and store name.

[0585] Input: Extracted text data

[0586] Data processing: Analyze text data using natural language processing techniques and extract important information (e.g., amount, store name).

[0587] Output: Structured data in JSON format (including important information)

[0588] Step 6:

[0589] The server sends the generated JSON format data to the terminal, where the user can check and modify it.

[0590] Input: Structured data in JSON format

[0591] Data processing: Data is sent to the device and displayed in a dedicated application. The user can check and edit the information.

[0592] Output: Data reviewed and corrected by the user

[0593] Step 7:

[0594] The server automatically generates a tax return based on the confirmed and corrected data and validates it internally.

[0595] Input: Data modified by the user

[0596] Data processing: Using generative AI models, tax returns are automatically generated and the data consistency is verified internally.

[0597] Output: Auto-generated tax return

[0598] Step 8:

[0599] The server submits the automatically generated tax return to the electronic filing system.

[0600] Input: Auto-generated tax return

[0601] Data processing: Send data based on the submission protocol to the electronic reporting system.

[0602] Output: Tax return submitted to the electronic filing system

[0603] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0604] The tax return assistant system of this invention combines an emotion engine to improve the user experience, allowing individual users to easily file tax returns. The system mainly works as follows: the user reads document data, uploads it to a server, the server analyzes and generates the data, and then the emotion engine recognizes the user's emotional state and allows the user to confirm and correct the data before automatically generating the tax return, which is finally submitted to the electronic filing system.

[0605] Technical configuration

[0606] 1. Acquisition of document data

[0607] The user uses the device to take an image of a receipt or certificate, which then captures the document data and temporarily stores it in the device's local storage.

[0608] 2. Uploading data

[0609] The user uploads the acquired document data to the server through a dedicated tax return assistant app. The device then sends the user-selected images to the server while maintaining appropriate security.

[0610] 3. Data analysis (image recognition)

[0611] The server receives the uploaded image data and extracts text from the document image using image processing technology such as OpenCV. A character recognition engine such as PyTesseract then extracts the text from the image as text data.

[0612] 4. Data Analysis (Natural Language Processing)

[0613] The server uses natural language processing technology to analyze the text data it acquires and extracts the information necessary for filing tax returns (e.g., amount, date, name of local government). During this process, the AI ​​algorithm identifies specific patterns and keywords, and categorizes and organizes the information.

[0614] 5. Emotion Recognition by Emotion Engine

[0615] An emotion engine installed on the server or device analyzes the user's facial image and voice data to recognize their emotional state. The emotional data acquired by the emotion engine is fed back to the entire system and used to improve the user experience. For example, if the user is under high stress, the interface design or messages can be changed.

[0616] 6. Information Generation and Verification

[0617] Based on the analyzed data and feedback from the emotion engine, the server generates data summarizing the information required for filing tax returns. This generated data is sent to the terminal and displayed to the user. The user checks the displayed information and manually corrects or adds anything necessary. The emotion engine monitors the user's emotional state throughout this process, displaying help information and advice as appropriate.

[0618] 7. Automatic generation of tax returns

[0619] The server automatically generates a tax return based on the final confirmed information, and the generated return is verified internally to ensure its accuracy.

[0620] 8. Submission to the electronic filing system

[0621] The server submits the generated tax return to the official electronic tax return system. Once the submission is complete, the server sends a notification to the user that the return has been completed. This notification can be sent via email or in-app notification.

[0622] Specific examples

[0623] For example, when a user wants to use a receipt for hometown tax donations, they take a photo of the receipt with their smartphone camera and upload the image to the server via the app. The server analyzes the image, extracts the amount and name of the local government, and then uses an emotion engine to recognize the user's emotional state. Based on the extracted data and emotion data, the most appropriate information and advice is displayed on the user's device. After the user confirms and corrects it, the server finally generates a tax return and submits it to the electronic filing system, completing the tax return.

[0624] In this way, the system of the present invention simplifies the complicated procedures for users and provides optimal support according to each individual's emotional state, thereby saving a great deal of time and effort.

[0625] The processing flow will be explained below.

[0626] Step 1: Scan your documents

[0627] The user uses the smartphone camera to take a photo of a document such as a receipt or certificate. The image is saved in the device's local storage. The user then launches the dedicated tax return assistant app and selects the image they took.

[0628] Step 2: Upload your data

[0629] The user uploads the saved document data to the server through the app. The device sends the selected image to the server using a secure communication protocol (such as SSL). The upload progress is displayed within the app.

[0630] Step 3: Data analysis (image recognition)

[0631] The server receives the uploaded image data. Next, it uses image processing technology such as OpenCV to extract characters from the image. Then it uses a character recognition engine such as PyTesseract to obtain the text in the image as text data.

[0632] Step 4: Data analysis (natural language processing)

[0633] The server analyzes the acquired text data using natural language processing technology. Specifically, it extracts important information from the text, such as amounts, dates, and names of local governments. During this process, an AI algorithm identifies specific patterns and keywords and organizes the information.

[0634] Step 5: Emotion Recognition with the Emotion Engine

[0635] The device's camera and microphone are used to capture the user's facial image and voice data, and an emotion engine installed on the device or server analyzes this data to identify the user's emotional state.

[0636] Step 6: Generate feedback

[0637] Based on the emotional data acquired by the emotion engine, the server generates feedback and advice to improve the user experience. For example, if the user is feeling stressed, a soothing message or problem-solving advice will be displayed on the device.

[0638] Step 7: Verify and correct information

[0639] The server generates the information necessary for filing tax returns based on the analysis results. This information is sent to the device in JSON format or similar and displayed to the user. The user checks the displayed information and makes corrections or adds additional information as necessary. The emotion engine also operates during this process, providing help and guidance as appropriate.

[0640] Step 8: Auto-generate tax returns

[0641] The server automatically generates a tax return based on the verified information, which is then internally verified to ensure accuracy.

[0642] Step 9: Submit to the electronic filing system

[0643] The server submits the final generated tax return to the official electronic tax return system. Once the submission is complete, the server sends a notification to the user that the return has been completed. This notification can be sent via email or in-app notification.

[0644] These steps allow users to file their tax returns accurately and quickly with minimal effort, and the emotional feedback engine improves the user experience, making the filing process more convenient.

[0645] Example 2

[0646] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0647] Traditional tax return systems require manual data entry and complex procedures, placing a heavy burden on users. Furthermore, due to the lack of emotional state recognition and feedback, tax return filing is often done under stress, increasing the likelihood of errors and delays in entry.

[0648] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0649] In this invention, the server includes: [means for analyzing data on the server and generating information necessary for filing tax returns;] [means for sending the generated information to the user's terminal so that the user can confirm and correct it; and] [means for acquiring the user's emotional state and feeding that data back to the entire system.] This makes it possible [to simplify the complicated tax return procedure and provide optimal support according to the user's emotional state].

[0650] "User" refers to an individual or corporation that uses the System to file tax returns.

[0651] "Document data" refers to data that has been photographed and stored digitally from physical documents such as receipts and certificates.

[0652] "Terminal" refers to a device such as a smartphone, tablet, or PC that is used to photograph, save, and upload document data to a server.

[0653] "Server" refers to the computer system that analyzes uploaded document data and generates and manages the information necessary for filing tax returns.

[0654] "Emotion engine" refers to artificial intelligence technology that analyzes a user's facial images and voice data to recognize their emotional state.

[0655] "Image processing technology" refers to computer vision technology used to extract character strings from document data (images).

[0656] "Natural language processing technology" refers to the technology used to analyze acquired text data and extract and classify information necessary for filing tax returns.

[0657] "Interface" refers to the user interface through which a user interacts with the system and makes any necessary corrections or additional inputs.

[0658] "Electronic filing system" means a system provided by a government or related authority for filing tax returns electronically.

[0659] "Automatic generation" refers to the process by which a server automatically generates tax returns based on specific rules and algorithms.

[0660] "Feedback" refers to the process by which the emotion engine makes adaptive adjustments to the entire system based on the user's emotional state.

[0661] MODE FOR CARRYING OUT THE INVENTION

[0662] The tax return assistant system of this invention is designed to allow individuals and corporations to easily file tax returns, and aims to improve the user experience by combining it with an emotion engine. The system mainly operates in a series of steps: the user reads document data, uploads it to a server, the server analyzes and generates the data, and then automatically generates a tax return after recognizing the user's emotional state using the emotion engine, and finally submits it to an electronic filing system.

[0663] Hardware and software used

[0664] Device: The user's smartphone, tablet, computer, etc., from which document data is acquired and uploaded to the server.

[0665] Server: Analyzes and generates data, recognizes emotions using an emotion engine, automatically generates tax returns, and submits them to the electronic tax return system.

[0666] Image processing technology: OpenCV is used to perform image recognition of document data.

[0667] Character recognition engine: Uses PyTesseract to extract characters in an image as text data.

[0668] Natural Language Processing (NLP): Uses technology to extract and organize information required for tax returns.

[0669] Emotion Engine: An artificial intelligence technology that analyzes a user's facial images and voice data to recognize their emotional state.

[0670] Specific example explanation

[0671] For example, consider the case where a user uses a receipt for hometown tax donations. The user takes a photo of the receipt using the camera on their smartphone. This document data is then uploaded to the server using the tax return assistant app. The server preprocesses the image using image processing technology (OpenCV) and extracts the text from the image using PyTesseract. It then uses natural language processing technology to identify necessary information such as the "amount" and "name of the local government" from the extracted text.

[0672] At the same time, the emotion engine analyzes the user's facial image and voice data to assess their emotional state (e.g., stress level). Based on this emotion data, the server provides the user with an optimal interface and displays necessary advice and help information.

[0673] Based on the analyzed data and emotional feedback, the information necessary for filing tax returns is generated and sent to the user's device. The user can then review the displayed information and make corrections or additional entries as necessary. During this process, the emotion engine monitors the user's emotional state and responds accordingly.

[0674] The server then automatically generates a tax return based on the final confirmed information and submits it to the electronic filing system. Once submission is complete, the server sends a notification of completion to the user via email or in-app notification.

[0675] Example prompts for generative AI models

[0676] "Please extract the amount and name of the local government from the hometown tax receipt, use an emotion engine to determine the user's stress level, and then automatically generate a tax return."

[0677] As described above, this system simplifies the complicated procedures users go through and provides optimal support according to their individual emotional state, thereby saving them a great deal of time and effort.

[0678] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0679] Program processing flow

[0680] Step 1:

[0681] The user takes a photo of the receipt or certificate required for tax return filing using a device such as a smartphone or PC. The captured image is temporarily saved in the device's local storage.

[0682] Specific behavior:

[0683] Input: receipt or certificate

[0684] Processing: Take a photo using the device's camera

[0685] Output: Image data saved in local storage

[0686] Step 2:

[0687] The user opens the Tax Return Assistant app and instructs it to upload the saved document data to the server. The device encrypts the image selected by the user and sends it to the server via a secure communication channel.

[0688] Specific behavior:

[0689] Input: Image data stored in local storage

[0690] Processing: Encryption of image data, transmission using HTTPS

[0691] Output: Image data uploaded to the server

[0692] Step 3:

[0693] The server analyzes the received image data using technologies such as OpenCV and PyTesseract, extracts character strings from the image, and saves them as text data.

[0694] Specific behavior:

[0695] Input: Image data uploaded to the server

[0696] Processing: Image preprocessing with OpenCV, character recognition with PyTesseract

[0697] Output: Text data

[0698] Step 4:

[0699] The server analyzes the acquired text data using natural language processing technology. The server identifies specific keywords and patterns, and extracts and organizes the necessary information (e.g., amount, date, name of local government).

[0700] Specific behavior:

[0701] Input: Text data

[0702] Processing: Applying NLP models to extract information

[0703] Output: Extracted information data (e.g. amount, date, name of local government)

[0704] Step 5:

[0705] The server or device uses an emotion engine to analyze the user's facial image and voice data to recognize the user's emotional state. The recognized emotional data is used throughout the system.

[0706] Specific behavior:

[0707] Input: User's facial image and voice data

[0708] Processing: Emotion recognition by emotion engine

[0709] Output: Emotion data

[0710] Step 6:

[0711] Based on the analyzed data and emotional feedback, the server generates optimal tax return information. The generated information is sent to the device, prompting the user to confirm and correct it. The emotion engine monitors the user's status throughout this process, providing help and advice as needed.

[0712] Specific behavior:

[0713] Input: Extracted information data, emotion data

[0714] Processing: Data formatting and generation

[0715] Output: Generated tax return information

[0716] Step 7:

[0717] The server automatically generates an official tax return based on the final verified information, which is then internally verified to ensure accuracy.

[0718] Specific behavior:

[0719] Input: Tax return information

[0720] Processing: Automatic generation and internal validation of tax returns

[0721] Output: Generated tax return

[0722] Step 8:

[0723] The server submits the verified tax return to the official electronic tax return system. Once the submission is complete, the server sends a notification to the user that the return has been completed. This notification can be sent via email or in-app notification.

[0724] Specific behavior:

[0725] Input: Generated tax return

[0726] Processing: Submission using the API of the electronic reporting system

[0727] Output: Notification of declaration completion

[0728] _Example of prompt sentence for generative AI model_

[0729] "Please extract the amount and name of the local government from the hometown tax receipt, use an emotion engine to determine the user's stress level, and then automatically generate a tax return."

[0730] (Application example 2)

[0731] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0732] In conventional tax return systems, users had to manually collect and analyze document data and generate tax returns, which was a cumbersome process that required time and effort. Furthermore, there was no mechanism to reduce the emotional burden caused by the stress and hassle, resulting in a poor user experience. Furthermore, there was no mechanism to generate and deliver interactive content based on the user's emotional state, making it difficult to personalize content based on the emotional state of each individual user.

[0733] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: [means for a user to read document data such as receipts and certificates;] [means for uploading the read data to the server;] [means for analyzing the data at the server and generating information necessary for filing a tax return;] [means for sending the generated information to the user's terminal so that the user can confirm and correct it;] [means for automatically generating a tax return based on the confirmed information and submitting it to an electronic filing system;] [means for optimizing the interface to the user based on the user's emotional state; and [means for generating story content based on the user's emotional data and delivering it interactively.] This simplifies the tax return process, reduces the emotional burden on the user, and enables personalized interactive content delivery based on the user's emotional state.

[0734] "User" refers to an individual who uses an application or system.

[0735] "Document data such as receipts and certificates" refers to digital data of official documents intended for the transfer of money or official certification.

[0736] "Server" refers to a computer system that stores, processes, and distributes data over a network.

[0737] "Analyzing data" refers to the process of processing acquired digital data and converting it into useful information.

[0738] "Information required for filing tax returns" refers to data such as amounts, dates, and personal information that are required when filing a tax return.

[0739] "Terminal" refers to a computer device such as a mobile phone, tablet, or PC that a user uses.

[0740] "Verify and correct" refers to the operation in which the user reviews the generated information and makes changes as necessary.

[0741] "Automatically generating tax returns" refers to the process by which the system automatically generates the necessary tax return documents based on the collected and analyzed data.

[0742] "Electronic filing system" refers to the official online platform for filing tax returns via the Internet.

[0743] "Emotional state" refers to data that indicates the user's mood or emotions at that time.

[0744] "Optimizing the interface" refers to improving the user experience by changing the system's display method and operating procedures according to the user's emotional state.

[0745] "Story content" refers to digital content in the form of a story that users can enjoy.

[0746] "Delivered interactively" means providing content in a format that allows users to progress through the story through choices.

[0747] The following description is given as a mode for carrying out the invention.

[0748] The present invention includes a tax return assistant system and an interactive story content distribution system. The system aims to enable individual users to complete tax return procedures easily and efficiently, and to provide personalized content based on the user's emotional state.

[0749] Hardware and Software

[0750] 1. Hardware

[0751] Devices: Smartphones, tablets, PCs, head-mounted displays, etc.

[0752] Server: A server capable of high-performance data processing

[0753] 2. Software

[0754] Image processing technology: OpenCV, PyTesseract

[0755] Natural language processing technology: NLTK, Transformers library

[0756] Emotion Recognition Engine: Emotion Recognition Model Using TensorFlow

[0757] Cloud storage: AWS S3

[0758] Web services: AWS Lambda, API Gateway

[0759] Details of data processing and calculation

[0760] 1. Obtaining and uploading document data

[0761] A user uses a device such as a smartphone to read document data such as receipts or certificates. The acquired data is temporarily stored in the device's local storage and then uploaded to a server via a dedicated application. This process uses the SSL / TLS protocol to maintain data confidentiality.

[0762] 2. Image Recognition and Text Data Extraction

[0763] The server uses OpenCV and PyTesseract to analyze uploaded document images and extract text data, such as the amount, date, and store name written on the receipt.

[0764] 3. Natural Language Processing of Text Data

[0765] The extracted text data is analyzed using NLTK and Transformers libraries. The server then accurately extracts and classifies the information required for tax returns. For example, it automatically assigns corresponding data to fields such as "amount," "date," and "address."

[0766] 4. Emotion Recognition and User Interface Optimization

[0767] An emotion recognition engine installed on the device or server analyzes the user's facial image and voice data to recognize their emotional state. This data is fed back into the interface design and help information display, helping to reduce the user's stress.

[0768] 5. Automatic generation and submission of tax return documents

[0769] The server automatically generates a tax return based on the data checked and corrected by the user. The generated return undergoes an internal validation process and is then submitted to the electronic filing system. After submission is complete, the server sends a completion notification to the user.

[0770] 6. Interactive story content generation and distribution

[0771] The server uses a generative AI model to generate story content based on the user's emotional state. Emotional data is used as prompts to dynamically generate story scenes. This content is delivered in an interactive format, allowing the user to progress through the story through choices.

[0772] Specific examples

[0773] For example, if a user is uploading image data for tax return and the process is stressful, the emotion recognition engine will detect this and provide a relaxing interface and friendly guidance for specific steps. After completing the tax return, the system will provide personalized story content to refresh the user.

[0774] Prompt Sentence Examples

[0775] Generate the following story scene based on the user's emotional data. The emotional state is excitement. The characters are a brave knight and a dragon. The location is a dark cave. The tone of the story is exciting and tense.

[0776] In this way, the system of the present invention can improve the user experience by simplifying the user's procedures and providing optimal support and entertainment tailored to their emotional state.

[0777] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0778] Step 1:

[0779] The user obtains document data such as receipts and certificates.

[0780] Specifically, the user takes an image of a document using a device such as a smartphone and temporarily stores it in the device's local storage.

[0781] Input: Document data (image format)

[0782] Output: Document data saved in local storage

[0783] Step 2:

[0784] The terminal uploads the document data to the server.

[0785] Specifically, the system uses a dedicated tax return assistant app to send the image selected by the user from the device to the server, while maintaining data confidentiality using the SSL / TLS protocol.

[0786] Input: Document data stored in local storage

[0787] Output: Document data uploaded to the server

[0788] Step 3:

[0789] The server analyzes the document data and performs image recognition.

[0790] Specifically, the server analyzes document images using OpenCV and PyTesseract and extracts text data.

[0791] Input: Document data uploaded to the server

[0792] Output: Extracted text data

[0793] Step 4:

[0794] The server analyzes the extracted text data using natural language processing.

[0795] Specifically, the server uses NLTK and Transformers libraries to analyze text data, extracting and classifying information necessary for tax returns. For example, it automatically identifies data corresponding to fields such as "amount," "date," and "address."

[0796] Input: Extracted text data

[0797] Output: Analyzed information (data required for tax return)

[0798] Step 5:

[0799] The server transmits the analyzed information to the user terminal.

[0800] Specifically, the information is sent to the user's device appropriately so that the user can check and modify it, and the interface is optimized to reduce user stress.

[0801] Input: Parsed information

[0802] Output: Information sent to the user's device

[0803] Step 6:

[0804] The terminal recognizes the user's emotional state.

[0805] Specifically, the system collects facial images and voice data using the device's camera and microphone, and determines the user's emotional state using an emotion recognition engine using TensorFlow.

[0806] Input: face image, voice data

[0807] Output: Emotional state data

[0808] Step 7:

[0809] A server generates story content based on emotional states.

[0810] Specifically, it uses emotion data as prompts to generate digital content (e.g., story scenes) using generative AI models, specifically generating exciting and suspenseful stories based on heightened emotional states.

[0811] Input: Emotional state data

[0812] Output: Generated story content

[0813] Step 8:

[0814] The server interactively delivers the generated story content to the user.

[0815] Specifically, the generated content is delivered to the user's device, allowing the user to progress through the story through choices. This delivery is done in real time, and the story changes dynamically depending on the user's choices.

[0816] Input: Generated story content

[0817] Output: Interactive story content delivered to user devices

[0818] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0819] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0820] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0821] [Third embodiment]

[0822] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0823] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0824] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0825] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0826] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0827] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0828] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0829] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0830] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0831] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0832] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0833] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0834] The tax return assistant system of this invention is designed to allow individual users to easily file tax returns. The system mainly works by having the user read document data, upload it to a server, analyze and generate the data, have the user check and correct it, automatically generate a tax return, and finally submit it to the electronic filing system.

[0835] Technical configuration

[0836] 1. Acquisition of document data

[0837] The user takes a photo of the receipt or certificate using a device such as a smartphone or computer. This captures the document data, which is then temporarily stored in the local storage of the user's device.

[0838] 2. Uploading data

[0839] The user uploads the acquired document data to the server through a dedicated tax return assistant app. The device then sends the user-selected images to the server while maintaining appropriate security.

[0840] 3. Data Analysis

[0841] The server analyzes the received document data using image processing and natural language processing technologies. First, it performs image recognition and extracts the text information written on the document as text. This extracted text data is then analyzed using natural language processing technology to identify important information such as the amount and name of the local government.

[0842] 4. Information Generation and Verification

[0843] Based on the parsed data, the server generates the information necessary for filing tax returns. This information is sent to the terminal as structured data in JSON format or similar and displayed to the user. The user can then check the displayed information and manually correct or add anything necessary.

[0844] 5. Automatic generation and submission of tax returns

[0845] The server automatically generates a tax return based on the final confirmed data. The generated tax return is then verified internally and then submitted to the electronic filing system. This series of processes allows users to easily file accurate tax returns.

[0846] Specific examples

[0847] For example, let's say a user uses a receipt for hometown tax donations. First, they take a photo of the receipt with their smartphone camera and upload the image to the server via the app. The server analyzes the image and extracts the amount and name of the local government. The extracted data is sent to the user's device, where the user can confirm and correct it. Finally, the server generates a tax return based on this information and submits it to the electronic filing system. This completes the tax return.

[0848] In this way, the system of the present invention allows users to simplify cumbersome procedures and save time and effort.

[0849] The processing flow will be explained below.

[0850] Step 1: Scan your documents

[0851] The user uses the camera function to take a photo of the document data and save it on the device. The user then launches a dedicated app and selects the image they took.

[0852] Step 2: Upload your data

[0853] The device sends the document data to the server. During transmission, the data is transferred securely using security protocols such as SSL. The upload progress is displayed on the user's device.

[0854] Step 3: Data analysis (image recognition)

[0855] The server receives the uploaded image data. Next, it uses image processing technology such as OpenCV to extract text from the document image. Then it uses a character recognition engine such as PyTesseract to recognize the text in the image and obtain the text data.

[0856] Step 4: Data analysis (natural language processing)

[0857] The server uses natural language processing technology to analyze the text data it acquires and extracts the information necessary for filing tax returns (e.g., amount, date, name of local government). During this process, the AI ​​algorithm identifies specific patterns and keywords, and categorizes and organizes the information.

[0858] Step 5: Generate and send information

[0859] Based on the analysis results, the server generates data summarizing the information required for filing, and this data is sent to the device in a structured format such as JSON.

[0860] Step 6: Verify and correct information

[0861] The device displays the received data to the user. The user can check the displayed information and manually edit or add anything if necessary. Once edits are complete, the user taps the "Confirm" button.

[0862] Step 7: Auto-generate tax returns

[0863] The server automatically generates tax returns based on the verified information, and the generated returns are validated internally to ensure accuracy.

[0864] Step 8: Submit to the electronic filing system

[0865] The server submits the generated tax return to the official electronic tax return system. Once the submission is complete, the server sends a notification to the user that the return has been completed. The notification can be sent via email or in-app notification.

[0866] These steps allow users to accurately file their tax returns with minimal effort on their part.

[0867] Example 1

[0868] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0869] Accounting and tax return procedures are extremely complicated and require a great deal of time and effort, especially for individual users. As a result, many users struggle to manage the necessary documents and prepare accurate tax returns. Furthermore, manual data entry and correction increases the risk of human error. Therefore, there is a need for a system that simplifies tax return procedures and processes them accurately and quickly.

[0870] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0871] In this invention, the server includes: [means using image processing technology to perform image recognition of document data]; [means using natural language processing technology to analyze text data extracted from the image]; and [means using JSON format to structuring data]. This enables users to easily digitize document data such as receipts and certificates, convert that data into information necessary for tax returns, automatically generate tax returns accurately and quickly, and submit them to the electronic filing system.

[0872] "User" refers to a person who uses the system to file a tax return.

[0873] "Document data" refers to digital image or text data, such as receipts and certificates, that contains information necessary for filing tax returns.

[0874] "Terminal" refers to an electronic device, such as a smartphone, PC, or tablet, that a user uses to acquire document data and send it to a server.

[0875] "Server" refers to the computer system that analyzes uploaded document data and generates and processes the necessary information.

[0876] "Analysis" refers to the process of extracting and classifying important information from uploaded document data.

[0877] "Image processing technology" refers to technology for extracting text information from image data, specifically using technologies such as OCR (optical character recognition).

[0878] "Natural language processing technology" refers to technology for analyzing extracted text data and identifying meaning and important information.

[0879] "JSON format" refers to a lightweight data exchange format that represents data in a structured manner and facilitates data exchange between systems.

[0880] "Electronic filing system" refers to an online system for submitting tax returns via the Internet.

[0881] "Interface" refers to the screen or operating means by which a user interacts with a system and inputs or modifies required information.

[0882] A "secure communication protocol" refers to a communication technology for encrypting data and transmitting it securely, specifically using SSL / TLS.

[0883] "Internal verification" refers to the process of verifying the accuracy of generated returns.

[0884] This invention is a system for easily filing tax returns, which automatically and efficiently performs a series of processes, such as allowing users to acquire document data, uploading it to a server, analyzing it, generating it, confirming it, and submitting it. Specific embodiments of this system are described below.

[0885] Acquiring document data

[0886] Users can take pictures of receipts and certificates using devices such as smartphones or PCs, and the images are saved in the device's local storage.

[0887] For example, when a user takes a photo of a hometown tax donation receipt with their smartphone camera, the image file is saved in the smartphone's "Pictures" folder.

[0888] Uploading data

[0889] The user launches the Tax Return Assistant app and selects the saved image file. The device then uses SSL / TLS encrypted communication to upload the image file to the server.

[0890] Specifically, when a user presses the "Upload" button within the app, the selected image is sent to the server via secure communication, using a secure communication protocol such as SSL / TLS.

[0891] Data analysis

[0892] The server uses image processing and natural language processing technologies to analyze the received image files. First, it uses an OCR (Optical Character Recognition) library as image processing technology to extract text information from the image. Next, it analyzes the extracted text data using a natural language processing library to identify important information such as the amount and the name of the local government.

[0893] Specifically, it uses Tesseract to extract text from images and spaCy, an advanced natural language processing library, to analyze the data.

[0894] Generate and verify information

[0895] The server generates the information necessary for filing tax returns based on the analysis results and sends that information in JSON format to the terminal. An interface is provided for the user to check the information displayed on the terminal screen and make corrections or additions as necessary.

[0896] For example, the information sent from the server can be displayed in a form within the app, and the "amount" field can be manually corrected.

[0897] Automatic generation and submission of tax returns

[0898] The server automatically generates tax returns based on the confirmed data. The generated tax returns undergo internal verification, and if there are no problems, they are submitted to the electronic tax return system. The entire process is automated, allowing users to easily file accurate tax returns.

[0899] For example, by linking to the e-Tax system, tax return data can be sent in CSV format, and the process of officially submitting the tax return can proceed.

[0900] Prompt Sentence Examples

[0901] Below are some example prompts to explain the details of this system to the generative AI model:

[0902] Prompt: Please explain the detailed process of the assistant system for easily filing tax returns. Please explain the specific steps involved in obtaining document data, uploading data, analyzing data, generating and verifying information, and automatically generating and submitting tax returns.

[0903] As a result, by using the system of the present invention, users can significantly reduce time and effort and efficiently file tax returns.

[0904] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0905] Step 1:

[0906] A user takes a picture of a receipt or certificate using the camera on their smartphone or computer. The input is the receipt or certificate, and the output is an image file. The captured image is saved in the device's local storage. Specifically, the user takes the picture using a camera app, and the image file is saved in, for example, a "Pictures" folder.

[0907] Step 2:

[0908] The user launches the Tax Return Assistant app and selects the saved image file. The input is the image file generated in step 1, and the output is the selected image file. Next, the device uploads the image file to the server using SSL / TLS. Specifically, the user taps the "Upload" button in the app, and the device sends the image to the server via encrypted communication.

[0909] Step 3:

[0910] The server analyzes the received image file. The input is the image file sent from the terminal, and the output is text data. First, the server uses the Tesseract library as an image processing technology to extract text information from the image. Specifically, the OCR process scans the image and converts the text information into text data.

[0911] Step 4:

[0912] The server analyzes the text data using natural language processing technology. The input is the text data extracted in step 3, and the output is structured data (e.g., amount, name of local government, date, etc.). Specifically, the server uses the spaCy library to analyze the text data and identify important information such as amount and name of local government.

[0913] Step 5:

[0914] The server generates the information required for tax returns in JSON format based on the analysis results and sends this JSON data to the terminal. The input is the structured data analyzed in step 4, and the output is JSON format data. Specifically, the server appropriately formats the analysis results and sends the data to the user's terminal.

[0915] Step 6:

[0916] The terminal displays the received JSON data on the application screen. The input is the JSON data sent from the server, and the output is an interface that allows the user to check and edit the data. The user uses this interface to check the data and manually edit or add data as needed. Specifically, the data is displayed on a form screen within the app, and the user operates it using the "Confirm" and "Edit" buttons.

[0917] Step 7:

[0918] The server automatically generates tax returns based on the confirmed and corrected data. The input is the data confirmed and corrected by the user, and the output is the tax return. Specifically, the server runs a script that creates a formatted tax return based on this data, and generates the return in a format such as PDF.

[0919] Step 8:

[0920] The server internally verifies the generated tax return and, if there are no problems, submits it to the electronic filing system. The input is the tax return generated in step 7, and the output is a notification of submission completion to the electronic filing system. Specifically, the server verifies the contents of the generated tax return and sends the data to an electronic filing system such as e-Tax for official submission.

[0921] These steps allow users to file their tax returns easily and accurately.

[0922] (Application example 1)

[0923] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0924] Modern individuals and business owners must manage numerous documents, such as receipts and certificates, on a daily basis, and they must accurately organize these documents, especially when filing tax returns. However, manually organizing these documents, extracting the necessary information, and preparing tax returns is a time-consuming and labor-intensive task. There is also a need for technology to efficiently record and manage purchase history when making purchases in stores. Therefore, there is a need for a system that automates these cumbersome and time-consuming tasks and allows users to easily and accurately file tax returns.

[0925] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0926] In this invention, the server includes: [means for a user to read document data such as receipts and certificates;] [means for uploading the read data to the server;] [means for analyzing the data on the server and generating information necessary for filing a tax return;] [means for sending the generated information to the user's terminal so the user can confirm and correct it;] [means for automatically generating a tax return based on the confirmed information and submitting it to an electronic filing system; and [means for using an application installed on a smartphone, smart glasses, or head-mounted display to easily manage the read data at the store via electronic device.] This allows users to easily manage receipt and document data, efficiently organize the information necessary for filing a tax return, and quickly and accurately generate and submit a tax return.

[0927] A "receipt" is a document issued to prove the amount paid in a transaction.

[0928] A "certificate" is an official document issued to certify a particular fact.

[0929] "Document data" refers to document information captured in digital form using a device such as a scanner or camera.

[0930] A "server" is a computer system that provides data and services over a network.

[0931] "Analysis" is the process of breaking down data and understanding and extracting its structure and meaning.

[0932] "Filing a tax return" is the process by which an individual or corporation reports the amount of tax due to tax authorities based on income and expenses for a certain period of time.

[0933] "Information generation" is the process of creating new information based on acquired and analyzed data.

[0934] "Transmission" is the process of moving data from one place to another.

[0935] "Confirmation" refers to the act of the user checking whether the generated information is correct.

[0936] "Correction" refers to the act of correcting any errors or missing information discovered as a result of verification.

[0937] A "tax return" is an official document that contains the information required to file a tax return.

[0938] "Automatic generation" is the process by which a system automatically creates documents and data without the need for manual input.

[0939] The "electronic filing system" is an online system for submitting tax returns to tax authorities via the Internet.

[0940] "Electronic devices" are digital devices such as smartphones, smart glasses, and head-mounted displays.

[0941] A "smartphone" is a multi-functional mobile device that can access the Internet and run applications in addition to the functions of a mobile phone.

[0942] "Smart glasses" are glasses-type mobile devices that can display information in front of the eyes.

[0943] A "head-mounted display" is a device that a user wears on their head to display visual information.

[0944] An "application" is a software program that runs on a computer to provide a particular function or service.

[0945] The system of this invention consists of three main elements: a user, a terminal, and a server. This system allows users to easily manage document data such as receipts and certificates, and efficiently generate and submit the information required for tax returns. The details of this system are described below.

[0946] Overall system configuration

[0947] 1. Users

[0948] Users acquire document data using electronic devices such as smartphones, smart glasses, and head-mounted displays (HMDs). For example, when a user wants to manage products purchased at a store, they acquire the data by taking a photo of the receipt with their smartphone. This document data is then temporarily stored in the local storage of the electronic device.

[0949] 2. Terminal

[0950] The terminal is responsible for uploading the document data acquired by the user to the server. The selected images are sent to the server while maintaining appropriate security through a dedicated tax return assistant application. This application is installed on a smartphone, smart glasses, or HMD.

[0951] 3. Server

[0952] The server analyzes the received document data and generates the information necessary for filing tax returns. Image recognition technology is used to extract text from the document data, and natural language processing technology is used to identify important information such as the amount and store name. The analysis results are sent to the terminal as structured data in JSON format or similar.

[0953] Program processing explanation

[0954] First, the user takes an image of the receipt using the smartphone camera. This data is saved in the smartphone's local storage. Next, the user launches the dedicated tax return assistant app and uploads the saved image data to the server. The uploaded data is analyzed by the server, and text information is extracted from the image.

[0955] The server analyzes the extracted text data using natural language processing technology to identify important information such as the amount and store name. The identified information is generated as structured data in JSON format and sent to the device. The user can then review the generated information on their device and manually correct or add additional information as necessary.

[0956] Finally, the server automatically generates a tax return based on the verified data and submits it to the electronic tax return system. Through this series of processes, users can easily and accurately file their tax returns.

[0957] Specific examples

[0958] For example, suppose a user goes shopping at a store and takes a photo of the receipt with their smartphone. This image is uploaded to the server via the app, and the server analyzes it. The analysis results include the amount and store name, and this information is sent to the device in JSON format. The user checks this information and makes any corrections, after which a final tax return is automatically generated. This tax return is then submitted to the electronic filing system.

[0959] Prompt Sentence Examples

[0960] "Please take a photo of your store receipts, organize the information, and automatically generate and provide your tax return."

[0961] This system allows users to automate cumbersome procedures and save a great deal of time and effort, making it a particularly useful tool for business people and freelancers to improve the efficiency of their daily work.

[0962] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0963] Step 1:

[0964] A user takes an image of a receipt or certificate using an electronic device such as a smartphone, using a dedicated application.

[0965] Input: Physical documents such as receipts and certificates and smartphone

[0966] Data processing: Use the smartphone's camera to capture digital images of documents.

[0967] Output: Image file of document data (e.g., JPEG format)

[0968] Step 2:

[0969] The device (smartphone) temporarily stores the captured image in local storage.

[0970] Input: Image file of photographed document data

[0971] Data processing: Image files are saved to the smartphone's local storage.

[0972] Output: Image file saved in local storage

[0973] Step 3:

[0974] The device (smartphone) uploads image data to the server via a dedicated app, with appropriate security measures taken during uploading.

[0975] Input: Image file stored in local storage

[0976] Data processing: Send the image file to the server using an HTTP POST request.

[0977] Output: Image file uploaded to the server

[0978] Step 4:

[0979] The server analyzes the received image data. First, it uses image recognition technology to extract text information from the image.

[0980] Input: Image file uploaded to the server

[0981] Data processing: Extract text from images using OCR (Optical Character Recognition) technology.

[0982] Output: Extracted text data

[0983] Step 5:

[0984] The server analyzes the extracted text data using natural language processing technology to identify important information such as the amount and store name.

[0985] Input: Extracted text data

[0986] Data processing: Analyze text data using natural language processing techniques and extract important information (e.g., amount, store name).

[0987] Output: Structured data in JSON format (including important information)

[0988] Step 6:

[0989] The server sends the generated JSON format data to the terminal, where the user can check and modify it.

[0990] Input: Structured data in JSON format

[0991] Data processing: Data is sent to the device and displayed in a dedicated application. The user can check and edit the information.

[0992] Output: Data reviewed and corrected by the user

[0993] Step 7:

[0994] The server automatically generates a tax return based on the confirmed and corrected data and validates it internally.

[0995] Input: Data modified by the user

[0996] Data processing: Using generative AI models, tax returns are automatically generated and the data consistency is verified internally.

[0997] Output: Auto-generated tax return

[0998] Step 8:

[0999] The server submits the automatically generated tax return to the electronic filing system.

[1000] Input: Auto-generated tax return

[1001] Data processing: Send data based on the submission protocol to the electronic reporting system.

[1002] Output: Tax return submitted to the electronic filing system

[1003] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1004] The tax return assistant system of this invention combines an emotion engine to improve the user experience, allowing individual users to easily file tax returns. The system mainly works as follows: the user reads document data, uploads it to a server, the server analyzes and generates the data, and then the emotion engine recognizes the user's emotional state and allows the user to confirm and correct the data before automatically generating the tax return, which is finally submitted to the electronic filing system.

[1005] Technical configuration

[1006] 1. Acquisition of document data

[1007] The user uses the device to take an image of a receipt or certificate, which then captures the document data and temporarily stores it in the device's local storage.

[1008] 2. Uploading data

[1009] The user uploads the acquired document data to the server through a dedicated tax return assistant app. The device then sends the user-selected images to the server while maintaining appropriate security.

[1010] 3. Data analysis (image recognition)

[1011] The server receives the uploaded image data and extracts text from the document image using image processing technology such as OpenCV. A character recognition engine such as PyTesseract then extracts the text from the image as text data.

[1012] 4. Data Analysis (Natural Language Processing)

[1013] The server uses natural language processing technology to analyze the text data it acquires and extracts the information necessary for filing tax returns (e.g., amount, date, name of local government). During this process, the AI ​​algorithm identifies specific patterns and keywords, and categorizes and organizes the information.

[1014] 5. Emotion Recognition by Emotion Engine

[1015] An emotion engine installed on the server or device analyzes the user's facial image and voice data to recognize their emotional state. The emotional data acquired by the emotion engine is fed back to the entire system and used to improve the user experience. For example, if the user is under high stress, the interface design or messages can be changed.

[1016] 6. Information Generation and Verification

[1017] Based on the analyzed data and feedback from the emotion engine, the server generates data summarizing the information required for filing tax returns. This generated data is sent to the terminal and displayed to the user. The user checks the displayed information and manually corrects or adds anything necessary. The emotion engine monitors the user's emotional state throughout this process, displaying help information and advice as appropriate.

[1018] 7. Automatic generation of tax returns

[1019] The server automatically generates a tax return based on the final confirmed information, and the generated return is verified internally to ensure its accuracy.

[1020] 8. Submission to the electronic filing system

[1021] The server submits the generated tax return to the official electronic tax return system. Once the submission is complete, the server sends a notification to the user that the return has been completed. This notification can be sent via email or in-app notification.

[1022] Specific examples

[1023] For example, when a user wants to use a receipt for hometown tax donations, they take a photo of the receipt with their smartphone camera and upload the image to the server via the app. The server analyzes the image, extracts the amount and name of the local government, and then uses an emotion engine to recognize the user's emotional state. Based on the extracted data and emotion data, the most appropriate information and advice is displayed on the user's device. After the user confirms and corrects it, the server finally generates a tax return and submits it to the electronic filing system, completing the tax return.

[1024] In this way, the system of the present invention simplifies the complicated procedures for users and provides optimal support according to each individual's emotional state, thereby saving a great deal of time and effort.

[1025] The processing flow will be explained below.

[1026] Step 1: Scan your documents

[1027] The user uses the smartphone camera to take a photo of a document such as a receipt or certificate. The image is saved in the device's local storage. The user then launches the dedicated tax return assistant app and selects the image they took.

[1028] Step 2: Upload your data

[1029] The user uploads the saved document data to the server through the app. The device sends the selected image to the server using a secure communication protocol (such as SSL). The upload progress is displayed within the app.

[1030] Step 3: Data analysis (image recognition)

[1031] The server receives the uploaded image data. Next, it uses image processing technology such as OpenCV to extract characters from the image. Then it uses a character recognition engine such as PyTesseract to obtain the text in the image as text data.

[1032] Step 4: Data analysis (natural language processing)

[1033] The server analyzes the acquired text data using natural language processing technology. Specifically, it extracts important information from the text, such as amounts, dates, and names of local governments. During this process, an AI algorithm identifies specific patterns and keywords and organizes the information.

[1034] Step 5: Emotion Recognition with the Emotion Engine

[1035] The device's camera and microphone are used to capture the user's facial image and voice data, and an emotion engine installed on the device or server analyzes this data to identify the user's emotional state.

[1036] Step 6: Generate feedback

[1037] Based on the emotional data acquired by the emotion engine, the server generates feedback and advice to improve the user experience. For example, if the user is feeling stressed, a soothing message or problem-solving advice will be displayed on the device.

[1038] Step 7: Verify and correct information

[1039] The server generates the information necessary for filing tax returns based on the analysis results. This information is sent to the device in JSON format or similar and displayed to the user. The user checks the displayed information and makes corrections or adds additional information as necessary. The emotion engine also operates during this process, providing help and guidance as appropriate.

[1040] Step 8: Auto-generate tax returns

[1041] The server automatically generates a tax return based on the verified information, which is then internally verified to ensure accuracy.

[1042] Step 9: Submit to the electronic filing system

[1043] The server submits the final generated tax return to the official electronic tax return system. Once the submission is complete, the server sends a notification to the user that the return has been completed. This notification can be sent via email or in-app notification.

[1044] These steps allow users to file their tax returns accurately and quickly with minimal effort, and the emotional feedback engine improves the user experience, making the filing process more convenient.

[1045] Example 2

[1046] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1047] Traditional tax return systems require manual data entry and complex procedures, placing a heavy burden on users. Furthermore, due to the lack of emotional state recognition and feedback, tax return filing is often done under stress, increasing the likelihood of errors and delays in entry.

[1048] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1049] In this invention, the server includes: [means for analyzing data on the server and generating information necessary for filing tax returns;] [means for sending the generated information to the user's terminal so that the user can confirm and correct it; and] [means for acquiring the user's emotional state and feeding that data back to the entire system.] This makes it possible [to simplify the complicated tax return procedure and provide optimal support according to the user's emotional state].

[1050] "User" refers to an individual or corporation that uses the System to file tax returns.

[1051] "Document data" refers to data that has been photographed and stored digitally from physical documents such as receipts and certificates.

[1052] "Terminal" refers to a device such as a smartphone, tablet, or PC that is used to photograph, save, and upload document data to a server.

[1053] "Server" refers to the computer system that analyzes uploaded document data and generates and manages the information necessary for filing tax returns.

[1054] "Emotion engine" refers to artificial intelligence technology that analyzes a user's facial images and voice data to recognize their emotional state.

[1055] "Image processing technology" refers to computer vision technology used to extract character strings from document data (images).

[1056] "Natural language processing technology" refers to the technology used to analyze acquired text data and extract and classify information necessary for filing tax returns.

[1057] "Interface" refers to the user interface through which a user interacts with the system and makes any necessary corrections or additional inputs.

[1058] "Electronic filing system" means a system provided by a government or related authority for filing tax returns electronically.

[1059] "Automatic generation" refers to the process by which a server automatically generates tax returns based on specific rules and algorithms.

[1060] "Feedback" refers to the process by which the emotion engine makes adaptive adjustments to the entire system based on the user's emotional state.

[1061] MODE FOR CARRYING OUT THE INVENTION

[1062] The tax return assistant system of this invention is designed to allow individuals and corporations to easily file tax returns, and aims to improve the user experience by combining it with an emotion engine. The system mainly operates in a series of steps: the user reads document data, uploads it to a server, the server analyzes and generates the data, and then automatically generates a tax return after recognizing the user's emotional state using the emotion engine, and finally submits it to an electronic filing system.

[1063] Hardware and software used

[1064] Device: The user's smartphone, tablet, computer, etc., from which document data is acquired and uploaded to the server.

[1065] Server: Analyzes and generates data, recognizes emotions using an emotion engine, automatically generates tax returns, and submits them to the electronic tax return system.

[1066] Image processing technology: OpenCV is used to perform image recognition of document data.

[1067] Character recognition engine: Uses PyTesseract to extract characters in an image as text data.

[1068] Natural Language Processing (NLP): Uses technology to extract and organize information required for tax returns.

[1069] Emotion Engine: An artificial intelligence technology that analyzes a user's facial images and voice data to recognize their emotional state.

[1070] Specific example explanation

[1071] For example, consider the case where a user uses a receipt for hometown tax donations. The user takes a photo of the receipt using the camera on their smartphone. This document data is then uploaded to the server using the tax return assistant app. The server preprocesses the image using image processing technology (OpenCV) and extracts the text from the image using PyTesseract. It then uses natural language processing technology to identify necessary information such as the "amount" and "name of the local government" from the extracted text.

[1072] At the same time, the emotion engine analyzes the user's facial image and voice data to assess their emotional state (e.g., stress level). Based on this emotion data, the server provides the user with an optimal interface and displays necessary advice and help information.

[1073] Based on the analyzed data and emotional feedback, the information necessary for filing tax returns is generated and sent to the user's device. The user can then review the displayed information and make corrections or additional entries as necessary. During this process, the emotion engine monitors the user's emotional state and responds accordingly.

[1074] The server then automatically generates a tax return based on the final confirmed information and submits it to the electronic filing system. Once submission is complete, the server sends a notification of completion to the user via email or in-app notification.

[1075] Example prompts for generative AI models

[1076] "Please extract the amount and name of the local government from the hometown tax receipt, use an emotion engine to determine the user's stress level, and then automatically generate a tax return."

[1077] As described above, this system simplifies the complicated procedures users go through and provides optimal support according to their individual emotional state, thereby saving them a great deal of time and effort.

[1078] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1079] Program processing flow

[1080] Step 1:

[1081] The user takes a photo of the receipt or certificate required for tax return filing using a device such as a smartphone or PC. The captured image is temporarily saved in the device's local storage.

[1082] Specific behavior:

[1083] Input: receipt or certificate

[1084] Processing: Take a photo using the device's camera

[1085] Output: Image data saved in local storage

[1086] Step 2:

[1087] The user opens the Tax Return Assistant app and instructs it to upload the saved document data to the server. The device encrypts the image selected by the user and sends it to the server via a secure communication channel.

[1088] Specific behavior:

[1089] Input: Image data stored in local storage

[1090] Processing: Encryption of image data, transmission using HTTPS

[1091] Output: Image data uploaded to the server

[1092] Step 3:

[1093] The server analyzes the received image data using technologies such as OpenCV and PyTesseract, extracts character strings from the image, and saves them as text data.

[1094] Specific behavior:

[1095] Input: Image data uploaded to the server

[1096] Processing: Image preprocessing with OpenCV, character recognition with PyTesseract

[1097] Output: Text data

[1098] Step 4:

[1099] The server analyzes the acquired text data using natural language processing technology. The server identifies specific keywords and patterns, and extracts and organizes the necessary information (e.g., amount, date, name of local government).

[1100] Specific behavior:

[1101] Input: Text data

[1102] Processing: Applying NLP models to extract information

[1103] Output: Extracted information data (e.g. amount, date, name of local government)

[1104] Step 5:

[1105] The server or device uses an emotion engine to analyze the user's facial image and voice data to recognize the user's emotional state. The recognized emotional data is used throughout the system.

[1106] Specific behavior:

[1107] Input: User's facial image and voice data

[1108] Processing: Emotion recognition by emotion engine

[1109] Output: Emotion data

[1110] Step 6:

[1111] Based on the analyzed data and emotional feedback, the server generates optimal tax return information. The generated information is sent to the device, prompting the user to confirm and correct it. The emotion engine monitors the user's status throughout this process, providing help and advice as needed.

[1112] Specific behavior:

[1113] Input: Extracted information data, emotion data

[1114] Processing: Data formatting and generation

[1115] Output: Generated tax return information

[1116] Step 7:

[1117] The server automatically generates an official tax return based on the final verified information, which is then internally verified to ensure accuracy.

[1118] Specific behavior:

[1119] Input: Tax return information

[1120] Processing: Automatic generation and internal validation of tax returns

[1121] Output: Generated tax return

[1122] Step 8:

[1123] The server submits the verified tax return to the official electronic tax return system. Once the submission is complete, the server sends a notification to the user that the return has been completed. This notification can be sent via email or in-app notification.

[1124] Specific behavior:

[1125] Input: Generated tax return

[1126] Processing: Submission using the API of the electronic reporting system

[1127] Output: Notification of declaration completion

[1128] _Example of prompt sentence for generative AI model_

[1129] "Please extract the amount and name of the local government from the hometown tax receipt, use an emotion engine to determine the user's stress level, and then automatically generate a tax return."

[1130] (Application example 2)

[1131] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1132] In conventional tax return systems, users had to manually collect and analyze document data and generate tax returns, which was a cumbersome process that required time and effort. Furthermore, there was no mechanism to reduce the emotional burden caused by the stress and hassle, resulting in a poor user experience. Furthermore, there was no mechanism to generate and deliver interactive content based on the user's emotional state, making it difficult to personalize content based on the emotional state of each individual user.

[1133] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: [means for a user to read document data such as receipts and certificates;] [means for uploading the read data to the server;] [means for analyzing the data at the server and generating information necessary for filing a tax return;] [means for sending the generated information to the user's terminal so that the user can confirm and correct it;] [means for automatically generating a tax return based on the confirmed information and submitting it to an electronic filing system;] [means for optimizing the interface to the user based on the user's emotional state; and [means for generating story content based on the user's emotional data and delivering it interactively.] This simplifies the tax return process, reduces the emotional burden on the user, and enables personalized interactive content delivery based on the user's emotional state.

[1134] "User" refers to an individual who uses an application or system.

[1135] "Document data such as receipts and certificates" refers to digital data of official documents intended for the transfer of money or official certification.

[1136] "Server" refers to a computer system that stores, processes, and distributes data over a network.

[1137] "Analyzing data" refers to the process of processing acquired digital data and converting it into useful information.

[1138] "Information required for filing tax returns" refers to data such as amounts, dates, and personal information that are required when filing a tax return.

[1139] "Terminal" refers to a computer device such as a mobile phone, tablet, or PC that a user uses.

[1140] "Verify and correct" refers to the operation in which the user reviews the generated information and makes changes as necessary.

[1141] "Automatically generating tax returns" refers to the process by which the system automatically generates the necessary tax return documents based on the collected and analyzed data.

[1142] "Electronic filing system" refers to the official online platform for filing tax returns via the Internet.

[1143] "Emotional state" refers to data that indicates the user's mood or emotions at that time.

[1144] "Optimizing the interface" refers to improving the user experience by changing the system's display method and operating procedures according to the user's emotional state.

[1145] "Story content" refers to digital content in the form of a story that users can enjoy.

[1146] "Delivered interactively" means providing content in a format that allows users to progress through the story through choices.

[1147] The following description is given as a mode for carrying out the invention.

[1148] The present invention includes a tax return assistant system and an interactive story content distribution system. The system aims to enable individual users to complete tax return procedures easily and efficiently, and to provide personalized content based on the user's emotional state.

[1149] Hardware and Software

[1150] 1. Hardware

[1151] Devices: Smartphones, tablets, PCs, head-mounted displays, etc.

[1152] Server: A server capable of high-performance data processing

[1153] 2. Software

[1154] Image processing technology: OpenCV, PyTesseract

[1155] Natural language processing technology: NLTK, Transformers library

[1156] Emotion Recognition Engine: Emotion Recognition Model Using TensorFlow

[1157] Cloud storage: AWS S3

[1158] Web services: AWS Lambda, API Gateway

[1159] Details of data processing and calculation

[1160] 1. Obtaining and uploading document data

[1161] A user uses a device such as a smartphone to read document data such as receipts or certificates. The acquired data is temporarily stored in the device's local storage and then uploaded to a server via a dedicated application. This process uses the SSL / TLS protocol to maintain data confidentiality.

[1162] 2. Image Recognition and Text Data Extraction

[1163] The server uses OpenCV and PyTesseract to analyze uploaded document images and extract text data, such as the amount, date, and store name written on the receipt.

[1164] 3. Natural Language Processing of Text Data

[1165] The extracted text data is analyzed using NLTK and Transformers libraries. The server then accurately extracts and classifies the information required for tax returns. For example, it automatically assigns corresponding data to fields such as "amount," "date," and "address."

[1166] 4. Emotion Recognition and User Interface Optimization

[1167] An emotion recognition engine installed on the device or server analyzes the user's facial image and voice data to recognize their emotional state. This data is fed back into the interface design and help information display, helping to reduce the user's stress.

[1168] 5. Automatic generation and submission of tax return documents

[1169] The server automatically generates a tax return based on the data checked and corrected by the user. The generated return undergoes an internal validation process and is then submitted to the electronic filing system. After submission is complete, the server sends a completion notification to the user.

[1170] 6. Interactive story content generation and distribution

[1171] The server uses a generative AI model to generate story content based on the user's emotional state. Emotional data is used as prompts to dynamically generate story scenes. This content is delivered in an interactive format, allowing the user to progress through the story through choices.

[1172] Specific examples

[1173] For example, if a user is uploading image data for tax return and the process is stressful, the emotion recognition engine will detect this and provide a relaxing interface and friendly guidance for specific steps. After completing the tax return, the system will provide personalized story content to refresh the user.

[1174] Prompt Sentence Examples

[1175] Generate the following story scene based on the user's emotional data. The emotional state is excitement. The characters are a brave knight and a dragon. The location is a dark cave. The tone of the story is exciting and tense.

[1176] In this way, the system of the present invention can improve the user experience by simplifying the user's procedures and providing optimal support and entertainment tailored to their emotional state.

[1177] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1178] Step 1:

[1179] The user obtains document data such as receipts and certificates.

[1180] Specifically, the user takes an image of a document using a device such as a smartphone and temporarily stores it in the device's local storage.

[1181] Input: Document data (image format)

[1182] Output: Document data saved in local storage

[1183] Step 2:

[1184] The terminal uploads the document data to the server.

[1185] Specifically, the system uses a dedicated tax return assistant app to send the image selected by the user from the device to the server, while maintaining data confidentiality using the SSL / TLS protocol.

[1186] Input: Document data stored in local storage

[1187] Output: Document data uploaded to the server

[1188] Step 3:

[1189] The server analyzes the document data and performs image recognition.

[1190] Specifically, the server analyzes document images using OpenCV and PyTesseract and extracts text data.

[1191] Input: Document data uploaded to the server

[1192] Output: Extracted text data

[1193] Step 4:

[1194] The server analyzes the extracted text data using natural language processing.

[1195] Specifically, the server uses NLTK and Transformers libraries to analyze text data, extracting and classifying information necessary for tax returns. For example, it automatically identifies data corresponding to fields such as "amount," "date," and "address."

[1196] Input: Extracted text data

[1197] Output: Analyzed information (data required for tax return)

[1198] Step 5:

[1199] The server transmits the analyzed information to the user terminal.

[1200] Specifically, the information is sent to the user's device appropriately so that the user can check and modify it, and the interface is optimized to reduce user stress.

[1201] Input: Parsed information

[1202] Output: Information sent to the user's device

[1203] Step 6:

[1204] The terminal recognizes the user's emotional state.

[1205] Specifically, the system collects facial images and voice data using the device's camera and microphone, and determines the user's emotional state using an emotion recognition engine using TensorFlow.

[1206] Input: face image, voice data

[1207] Output: Emotional state data

[1208] Step 7:

[1209] A server generates story content based on emotional states.

[1210] Specifically, it uses emotion data as prompts to generate digital content (e.g., story scenes) using generative AI models, specifically generating exciting and suspenseful stories based on heightened emotional states.

[1211] Input: Emotional state data

[1212] Output: Generated story content

[1213] Step 8:

[1214] The server interactively delivers the generated story content to the user.

[1215] Specifically, the generated content is delivered to the user's device, allowing the user to progress through the story through choices. This delivery is done in real time, and the story changes dynamically depending on the user's choices.

[1216] Input: Generated story content

[1217] Output: Interactive story content delivered to user devices

[1218] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1219] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1220] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1221] [Fourth embodiment]

[1222] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1223] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1224] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1225] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1226] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1227] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1228] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1229] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1230] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1231] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1232] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1233] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1234] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1235] The tax return assistant system of this invention is designed to allow individual users to easily file tax returns. The system mainly works by having the user read document data, upload it to a server, analyze and generate the data, have the user check and correct it, automatically generate a tax return, and finally submit it to the electronic filing system.

[1236] Technical configuration

[1237] 1. Acquisition of document data

[1238] The user takes a photo of the receipt or certificate using a device such as a smartphone or computer. This captures the document data, which is then temporarily stored in the local storage of the user's device.

[1239] 2. Uploading data

[1240] The user uploads the acquired document data to the server through a dedicated tax return assistant app. The device then sends the user-selected images to the server while maintaining appropriate security.

[1241] 3. Data Analysis

[1242] The server analyzes the received document data using image processing and natural language processing technologies. First, it performs image recognition and extracts the text information written on the document as text. This extracted text data is then analyzed using natural language processing technology to identify important information such as the amount and name of the local government.

[1243] 4. Information Generation and Verification

[1244] Based on the parsed data, the server generates the information necessary for filing tax returns. This information is sent to the terminal as structured data in JSON format or similar and displayed to the user. The user can then check the displayed information and manually correct or add anything necessary.

[1245] 5. Automatic generation and submission of tax returns

[1246] The server automatically generates a tax return based on the final confirmed data. The generated tax return is then verified internally and then submitted to the electronic filing system. This series of processes allows users to easily file accurate tax returns.

[1247] Specific examples

[1248] For example, let's say a user uses a receipt for hometown tax donations. First, they take a photo of the receipt with their smartphone camera and upload the image to the server via the app. The server analyzes the image and extracts the amount and name of the local government. The extracted data is sent to the user's device, where the user can confirm and correct it. Finally, the server generates a tax return based on this information and submits it to the electronic filing system. This completes the tax return.

[1249] In this way, the system of the present invention allows users to simplify cumbersome procedures and save time and effort.

[1250] The processing flow will be explained below.

[1251] Step 1: Scan your documents

[1252] The user uses the camera function to take a photo of the document data and save it on the device. The user then launches a dedicated app and selects the image they took.

[1253] Step 2: Upload your data

[1254] The device sends the document data to the server. During transmission, the data is transferred securely using security protocols such as SSL. The upload progress is displayed on the user's device.

[1255] Step 3: Data analysis (image recognition)

[1256] The server receives the uploaded image data. Next, it uses image processing technology such as OpenCV to extract text from the document image. Then it uses a character recognition engine such as PyTesseract to recognize the text in the image and obtain the text data.

[1257] Step 4: Data analysis (natural language processing)

[1258] The server uses natural language processing technology to analyze the text data it acquires and extracts the information necessary for filing tax returns (e.g., amount, date, name of local government). During this process, the AI ​​algorithm identifies specific patterns and keywords, and categorizes and organizes the information.

[1259] Step 5: Generate and send information

[1260] Based on the analysis results, the server generates data summarizing the information required for filing, and this data is sent to the device in a structured format such as JSON.

[1261] Step 6: Verify and correct information

[1262] The device displays the received data to the user. The user can check the displayed information and manually edit or add anything if necessary. Once edits are complete, the user taps the "Confirm" button.

[1263] Step 7: Auto-generate tax returns

[1264] The server automatically generates tax returns based on the verified information, and the generated returns are validated internally to ensure accuracy.

[1265] Step 8: Submit to the electronic filing system

[1266] The server submits the generated tax return to the official electronic tax return system. Once the submission is complete, the server sends a notification to the user that the return has been completed. The notification can be sent via email or in-app notification.

[1267] These steps allow users to accurately file their tax returns with minimal effort on their part.

[1268] Example 1

[1269] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1270] Accounting and tax return procedures are extremely complicated and require a great deal of time and effort, especially for individual users. As a result, many users struggle to manage the necessary documents and prepare accurate tax returns. Furthermore, manual data entry and correction increases the risk of human error. Therefore, there is a need for a system that simplifies tax return procedures and processes them accurately and quickly.

[1271] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1272] In this invention, the server includes: [means using image processing technology to perform image recognition of document data]; [means using natural language processing technology to analyze text data extracted from the image]; and [means using JSON format to structuring data]. This enables users to easily digitize document data such as receipts and certificates, convert that data into information necessary for tax returns, automatically generate tax returns accurately and quickly, and submit them to the electronic filing system.

[1273] "User" refers to a person who uses the system to file a tax return.

[1274] "Document data" refers to digital image or text data, such as receipts and certificates, that contains information necessary for filing tax returns.

[1275] "Terminal" refers to an electronic device, such as a smartphone, PC, or tablet, that a user uses to acquire document data and send it to a server.

[1276] "Server" refers to the computer system that analyzes uploaded document data and generates and processes the necessary information.

[1277] "Analysis" refers to the process of extracting and classifying important information from uploaded document data.

[1278] "Image processing technology" refers to technology for extracting text information from image data, specifically using technologies such as OCR (optical character recognition).

[1279] "Natural language processing technology" refers to technology for analyzing extracted text data and identifying meaning and important information.

[1280] "JSON format" refers to a lightweight data exchange format that represents data in a structured manner and facilitates data exchange between systems.

[1281] "Electronic filing system" refers to an online system for submitting tax returns via the Internet.

[1282] "Interface" refers to the screen or operating means by which a user interacts with a system and inputs or modifies required information.

[1283] A "secure communication protocol" refers to a communication technology for encrypting data and transmitting it securely, specifically using SSL / TLS.

[1284] "Internal verification" refers to the process of verifying the accuracy of generated returns.

[1285] This invention is a system for easily filing tax returns, which automatically and efficiently performs a series of processes, such as allowing users to acquire document data, uploading it to a server, analyzing it, generating it, confirming it, and submitting it. Specific embodiments of this system are described below.

[1286] Acquiring document data

[1287] Users can take pictures of receipts and certificates using devices such as smartphones or PCs, and the images are saved in the device's local storage.

[1288] For example, when a user takes a photo of a hometown tax donation receipt with their smartphone camera, the image file is saved in the smartphone's "Pictures" folder.

[1289] Uploading data

[1290] The user launches the Tax Return Assistant app and selects the saved image file. The device then uses SSL / TLS encrypted communication to upload the image file to the server.

[1291] Specifically, when a user presses the "Upload" button within the app, the selected image is sent to the server via secure communication, using a secure communication protocol such as SSL / TLS.

[1292] Data analysis

[1293] The server uses image processing and natural language processing technologies to analyze the received image files. First, it uses an OCR (Optical Character Recognition) library as image processing technology to extract text information from the image. Next, it analyzes the extracted text data using a natural language processing library to identify important information such as the amount and the name of the local government.

[1294] Specifically, it uses Tesseract to extract text from images and spaCy, an advanced natural language processing library, to analyze the data.

[1295] Generate and verify information

[1296] The server generates the information necessary for filing tax returns based on the analysis results and sends that information in JSON format to the terminal. An interface is provided for the user to check the information displayed on the terminal screen and make corrections or additions as necessary.

[1297] For example, the information sent from the server can be displayed in a form within the app, and the "amount" field can be manually corrected.

[1298] Automatic generation and submission of tax returns

[1299] The server automatically generates tax returns based on the confirmed data. The generated tax returns undergo internal verification, and if there are no problems, they are submitted to the electronic tax return system. The entire process is automated, allowing users to easily file accurate tax returns.

[1300] For example, by linking to the e-Tax system, tax return data can be sent in CSV format, and the process of officially submitting the tax return can proceed.

[1301] Prompt Sentence Examples

[1302] Below are some example prompts to explain the details of this system to the generative AI model:

[1303] Prompt: Please explain the detailed process of the assistant system for easily filing tax returns. Please explain the specific steps involved in obtaining document data, uploading data, analyzing data, generating and verifying information, and automatically generating and submitting tax returns.

[1304] As a result, by using the system of the present invention, users can significantly reduce time and effort and efficiently file tax returns.

[1305] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1306] Step 1:

[1307] A user takes a picture of a receipt or certificate using the camera on their smartphone or computer. The input is the receipt or certificate, and the output is an image file. The captured image is saved in the device's local storage. Specifically, the user takes the picture using a camera app, and the image file is saved in, for example, a "Pictures" folder.

[1308] Step 2:

[1309] The user launches the Tax Return Assistant app and selects the saved image file. The input is the image file generated in step 1, and the output is the selected image file. Next, the device uploads the image file to the server using SSL / TLS. Specifically, the user taps the "Upload" button in the app, and the device sends the image to the server via encrypted communication.

[1310] Step 3:

[1311] The server analyzes the received image file. The input is the image file sent from the terminal, and the output is text data. First, the server uses the Tesseract library as an image processing technology to extract text information from the image. Specifically, the OCR process scans the image and converts the text information into text data.

[1312] Step 4:

[1313] The server analyzes the text data using natural language processing technology. The input is the text data extracted in step 3, and the output is structured data (e.g., amount, name of local government, date, etc.). Specifically, the server uses the spaCy library to analyze the text data and identify important information such as amount and name of local government.

[1314] Step 5:

[1315] The server generates the information required for tax returns in JSON format based on the analysis results and sends this JSON data to the terminal. The input is the structured data analyzed in step 4, and the output is JSON format data. Specifically, the server appropriately formats the analysis results and sends the data to the user's terminal.

[1316] Step 6:

[1317] The terminal displays the received JSON data on the application screen. The input is the JSON data sent from the server, and the output is an interface that allows the user to check and edit the data. The user uses this interface to check the data and manually edit or add data as needed. Specifically, the data is displayed on a form screen within the app, and the user operates it using the "Confirm" and "Edit" buttons.

[1318] Step 7:

[1319] The server automatically generates tax returns based on the confirmed and corrected data. The input is the data confirmed and corrected by the user, and the output is the tax return. Specifically, the server runs a script that creates a formatted tax return based on this data, and generates the return in a format such as PDF.

[1320] Step 8:

[1321] The server internally verifies the generated tax return and, if there are no problems, submits it to the electronic filing system. The input is the tax return generated in step 7, and the output is a notification of submission completion to the electronic filing system. Specifically, the server verifies the contents of the generated tax return and sends the data to an electronic filing system such as e-Tax for official submission.

[1322] These steps allow users to file their tax returns easily and accurately.

[1323] (Application example 1)

[1324] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1325] Modern individuals and business owners must manage numerous documents, such as receipts and certificates, on a daily basis, and they must accurately organize these documents, especially when filing tax returns. However, manually organizing these documents, extracting the necessary information, and preparing tax returns is a time-consuming and labor-intensive task. There is also a need for technology to efficiently record and manage purchase history when making purchases in stores. Therefore, there is a need for a system that automates these cumbersome and time-consuming tasks and allows users to easily and accurately file tax returns.

[1326] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1327] In this invention, the server includes: [means for a user to read document data such as receipts and certificates;] [means for uploading the read data to the server;] [means for analyzing the data on the server and generating information necessary for filing a tax return;] [means for sending the generated information to the user's terminal so the user can confirm and correct it;] [means for automatically generating a tax return based on the confirmed information and submitting it to an electronic filing system; and [means for using an application installed on a smartphone, smart glasses, or head-mounted display to easily manage the read data at the store via electronic device.] This allows users to easily manage receipt and document data, efficiently organize the information necessary for filing a tax return, and quickly and accurately generate and submit a tax return.

[1328] A "receipt" is a document issued to prove the amount paid in a transaction.

[1329] A "certificate" is an official document issued to certify a particular fact.

[1330] "Document data" refers to document information captured in digital form using a device such as a scanner or camera.

[1331] A "server" is a computer system that provides data and services over a network.

[1332] "Analysis" is the process of breaking down data and understanding and extracting its structure and meaning.

[1333] "Filing a tax return" is the process by which an individual or corporation reports the amount of tax due to tax authorities based on income and expenses for a certain period of time.

[1334] "Information generation" is the process of creating new information based on acquired and analyzed data.

[1335] "Transmission" is the process of moving data from one place to another.

[1336] "Confirmation" refers to the act of the user checking whether the generated information is correct.

[1337] "Correction" refers to the act of correcting any errors or missing information discovered as a result of verification.

[1338] A "tax return" is an official document that contains the information required to file a tax return.

[1339] "Automatic generation" is the process by which a system automatically creates documents and data without the need for manual input.

[1340] The "electronic filing system" is an online system for submitting tax returns to tax authorities via the Internet.

[1341] "Electronic devices" are digital devices such as smartphones, smart glasses, and head-mounted displays.

[1342] A "smartphone" is a multi-functional mobile device that can access the Internet and run applications in addition to the functions of a mobile phone.

[1343] "Smart glasses" are glasses-type mobile devices that can display information in front of the eyes.

[1344] A "head-mounted display" is a device that a user wears on their head to display visual information.

[1345] An "application" is a software program that runs on a computer to provide a particular function or service.

[1346] The system of this invention consists of three main elements: a user, a terminal, and a server. This system allows users to easily manage document data such as receipts and certificates, and efficiently generate and submit the information required for tax returns. The details of this system are described below.

[1347] Overall system configuration

[1348] 1. Users

[1349] Users acquire document data using electronic devices such as smartphones, smart glasses, and head-mounted displays (HMDs). For example, when a user wants to manage products purchased at a store, they acquire the data by taking a photo of the receipt with their smartphone. This document data is then temporarily stored in the local storage of the electronic device.

[1350] 2. Terminal

[1351] The terminal is responsible for uploading the document data acquired by the user to the server. The selected images are sent to the server while maintaining appropriate security through a dedicated tax return assistant application. This application is installed on a smartphone, smart glasses, or HMD.

[1352] 3. Server

[1353] The server analyzes the received document data and generates the information necessary for filing tax returns. Image recognition technology is used to extract text from the document data, and natural language processing technology is used to identify important information such as the amount and store name. The analysis results are sent to the terminal as structured data in JSON format or similar.

[1354] Program processing explanation

[1355] First, the user takes an image of the receipt using the smartphone camera. This data is saved in the smartphone's local storage. Next, the user launches the dedicated tax return assistant app and uploads the saved image data to the server. The uploaded data is analyzed by the server, and text information is extracted from the image.

[1356] The server analyzes the extracted text data using natural language processing technology to identify important information such as the amount and store name. The identified information is generated as structured data in JSON format and sent to the device. The user can then review the generated information on their device and manually correct or add additional information as necessary.

[1357] Finally, the server automatically generates a tax return based on the verified data and submits it to the electronic tax return system. Through this series of processes, users can easily and accurately file their tax returns.

[1358] Specific examples

[1359] For example, suppose a user goes shopping at a store and takes a photo of the receipt with their smartphone. This image is uploaded to the server via the app, and the server analyzes it. The analysis results include the amount and store name, and this information is sent to the device in JSON format. The user checks this information and makes any corrections, after which a final tax return is automatically generated. This tax return is then submitted to the electronic filing system.

[1360] Prompt Sentence Examples

[1361] "Please take a photo of your store receipts, organize the information, and automatically generate and provide your tax return."

[1362] This system allows users to automate cumbersome procedures and save a great deal of time and effort, making it a particularly useful tool for business people and freelancers to improve the efficiency of their daily work.

[1363] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1364] Step 1:

[1365] A user takes an image of a receipt or certificate using an electronic device such as a smartphone, using a dedicated application.

[1366] Input: Physical documents such as receipts and certificates and smartphone

[1367] Data processing: Use the smartphone's camera to capture digital images of documents.

[1368] Output: Image file of document data (e.g., JPEG format)

[1369] Step 2:

[1370] The device (smartphone) temporarily stores the captured image in local storage.

[1371] Input: Image file of photographed document data

[1372] Data processing: Image files are saved to the smartphone's local storage.

[1373] Output: Image file saved in local storage

[1374] Step 3:

[1375] The device (smartphone) uploads image data to the server via a dedicated app, with appropriate security measures taken during uploading.

[1376] Input: Image file stored in local storage

[1377] Data processing: Send the image file to the server using an HTTP POST request.

[1378] Output: Image file uploaded to the server

[1379] Step 4:

[1380] The server analyzes the received image data. First, it uses image recognition technology to extract text information from the image.

[1381] Input: Image file uploaded to the server

[1382] Data processing: Extract text from images using OCR (Optical Character Recognition) technology.

[1383] Output: Extracted text data

[1384] Step 5:

[1385] The server analyzes the extracted text data using natural language processing technology to identify important information such as the amount and store name.

[1386] Input: Extracted text data

[1387] Data processing: Analyze text data using natural language processing techniques and extract important information (e.g., amount, store name).

[1388] Output: Structured data in JSON format (including important information)

[1389] Step 6:

[1390] The server sends the generated JSON format data to the terminal, where the user can check and modify it.

[1391] Input: Structured data in JSON format

[1392] Data processing: Data is sent to the device and displayed in a dedicated application. The user can check and edit the information.

[1393] Output: Data reviewed and corrected by the user

[1394] Step 7:

[1395] The server automatically generates a tax return based on the confirmed and corrected data and validates it internally.

[1396] Input: Data modified by the user

[1397] Data processing: Using generative AI models, tax returns are automatically generated and the data consistency is verified internally.

[1398] Output: Auto-generated tax return

[1399] Step 8:

[1400] The server submits the automatically generated tax return to the electronic filing system.

[1401] Input: Auto-generated tax return

[1402] Data processing: Send data based on the submission protocol to the electronic reporting system.

[1403] Output: Tax return submitted to the electronic filing system

[1404] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1405] The tax return assistant system of this invention combines an emotion engine to improve the user experience, allowing individual users to easily file tax returns. The system mainly works as follows: the user reads document data, uploads it to a server, the server analyzes and generates the data, and then the emotion engine recognizes the user's emotional state and allows the user to confirm and correct the data before automatically generating the tax return, which is finally submitted to the electronic filing system.

[1406] Technical configuration

[1407] 1. Acquisition of document data

[1408] The user uses the device to take an image of a receipt or certificate, which then captures the document data and temporarily stores it in the device's local storage.

[1409] 2. Uploading data

[1410] The user uploads the acquired document data to the server through a dedicated tax return assistant app. The device then sends the user-selected images to the server while maintaining appropriate security.

[1411] 3. Data analysis (image recognition)

[1412] The server receives the uploaded image data and extracts text from the document image using image processing technology such as OpenCV. A character recognition engine such as PyTesseract then extracts the text from the image as text data.

[1413] 4. Data Analysis (Natural Language Processing)

[1414] The server uses natural language processing technology to analyze the text data it acquires and extracts the information necessary for filing tax returns (e.g., amount, date, name of local government). During this process, the AI ​​algorithm identifies specific patterns and keywords, and categorizes and organizes the information.

[1415] 5. Emotion Recognition by Emotion Engine

[1416] An emotion engine installed on the server or device analyzes the user's facial image and voice data to recognize their emotional state. The emotional data acquired by the emotion engine is fed back to the entire system and used to improve the user experience. For example, if the user is under high stress, the interface design or messages can be changed.

[1417] 6. Information Generation and Verification

[1418] Based on the analyzed data and feedback from the emotion engine, the server generates data summarizing the information required for filing tax returns. This generated data is sent to the terminal and displayed to the user. The user checks the displayed information and manually corrects or adds anything necessary. The emotion engine monitors the user's emotional state throughout this process, displaying help information and advice as appropriate.

[1419] 7. Automatic generation of tax returns

[1420] The server automatically generates a tax return based on the final confirmed information, and the generated return is verified internally to ensure its accuracy.

[1421] 8. Submission to the electronic filing system

[1422] The server submits the generated tax return to the official electronic tax return system. Once the submission is complete, the server sends a notification to the user that the return has been completed. This notification can be sent via email or in-app notification.

[1423] Specific examples

[1424] For example, when a user wants to use a receipt for hometown tax donations, they take a photo of the receipt with their smartphone camera and upload the image to the server via the app. The server analyzes the image, extracts the amount and name of the local government, and then uses an emotion engine to recognize the user's emotional state. Based on the extracted data and emotion data, the most appropriate information and advice is displayed on the user's device. After the user confirms and corrects it, the server finally generates a tax return and submits it to the electronic filing system, completing the tax return.

[1425] In this way, the system of the present invention simplifies the complicated procedures for users and provides optimal support according to each individual's emotional state, thereby saving a great deal of time and effort.

[1426] The processing flow will be explained below.

[1427] Step 1: Scan your documents

[1428] The user uses the smartphone camera to take a photo of a document such as a receipt or certificate. The image is saved in the device's local storage. The user then launches the dedicated tax return assistant app and selects the image they took.

[1429] Step 2: Upload your data

[1430] The user uploads the saved document data to the server through the app. The device sends the selected image to the server using a secure communication protocol (such as SSL). The upload progress is displayed within the app.

[1431] Step 3: Data analysis (image recognition)

[1432] The server receives the uploaded image data. Next, it uses image processing technology such as OpenCV to extract characters from the image. Then it uses a character recognition engine such as PyTesseract to obtain the text in the image as text data.

[1433] Step 4: Data analysis (natural language processing)

[1434] The server analyzes the acquired text data using natural language processing technology. Specifically, it extracts important information from the text, such as amounts, dates, and names of local governments. During this process, an AI algorithm identifies specific patterns and keywords and organizes the information.

[1435] Step 5: Emotion Recognition with the Emotion Engine

[1436] The device's camera and microphone are used to capture the user's facial image and voice data, and an emotion engine installed on the device or server analyzes this data to identify the user's emotional state.

[1437] Step 6: Generate feedback

[1438] Based on the emotional data acquired by the emotion engine, the server generates feedback and advice to improve the user experience. For example, if the user is feeling stressed, a soothing message or problem-solving advice will be displayed on the device.

[1439] Step 7: Verify and correct information

[1440] The server generates the information necessary for filing tax returns based on the analysis results. This information is sent to the device in JSON format or similar and displayed to the user. The user checks the displayed information and makes corrections or adds additional information as necessary. The emotion engine also operates during this process, providing help and guidance as appropriate.

[1441] Step 8: Auto-generate tax returns

[1442] The server automatically generates a tax return based on the verified information, which is then internally verified to ensure accuracy.

[1443] Step 9: Submit to the electronic filing system

[1444] The server submits the final generated tax return to the official electronic tax return system. Once the submission is complete, the server sends a notification to the user that the return has been completed. This notification can be sent via email or in-app notification.

[1445] These steps allow users to file their tax returns accurately and quickly with minimal effort, and the emotional feedback engine improves the user experience, making the filing process more convenient.

[1446] Example 2

[1447] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1448] Traditional tax return systems require manual data entry and complex procedures, placing a heavy burden on users. Furthermore, due to the lack of emotional state recognition and feedback, tax return filing is often done under stress, increasing the likelihood of errors and delays in entry.

[1449] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1450] In this invention, the server includes: [means for analyzing data on the server and generating information necessary for filing tax returns;] [means for sending the generated information to the user's terminal so that the user can confirm and correct it; and] [means for acquiring the user's emotional state and feeding that data back to the entire system.] This makes it possible [to simplify the complicated tax return procedure and provide optimal support according to the user's emotional state].

[1451] "User" refers to an individual or corporation that uses the System to file tax returns.

[1452] "Document data" refers to data that has been photographed and stored digitally from physical documents such as receipts and certificates.

[1453] "Terminal" refers to a device such as a smartphone, tablet, or PC that is used to photograph, save, and upload document data to a server.

[1454] "Server" refers to the computer system that analyzes uploaded document data and generates and manages the information necessary for filing tax returns.

[1455] "Emotion engine" refers to artificial intelligence technology that analyzes a user's facial images and voice data to recognize their emotional state.

[1456] "Image processing technology" refers to computer vision technology used to extract character strings from document data (images).

[1457] "Natural language processing technology" refers to the technology used to analyze acquired text data and extract and classify information necessary for filing tax returns.

[1458] "Interface" refers to the user interface through which a user interacts with the system and makes any necessary corrections or additional inputs.

[1459] "Electronic filing system" means a system provided by a government or related authority for filing tax returns electronically.

[1460] "Automatic generation" refers to the process by which a server automatically generates tax returns based on specific rules and algorithms.

[1461] "Feedback" refers to the process by which the emotion engine makes adaptive adjustments to the entire system based on the user's emotional state.

[1462] MODE FOR CARRYING OUT THE INVENTION

[1463] The tax return assistant system of this invention is designed to allow individuals and corporations to easily file tax returns, and aims to improve the user experience by combining it with an emotion engine. The system mainly operates in a series of steps: the user reads document data, uploads it to a server, the server analyzes and generates the data, and then automatically generates a tax return after recognizing the user's emotional state using the emotion engine, and finally submits it to an electronic filing system.

[1464] Hardware and software used

[1465] Device: The user's smartphone, tablet, computer, etc., from which document data is acquired and uploaded to the server.

[1466] Server: Analyzes and generates data, recognizes emotions using an emotion engine, automatically generates tax returns, and submits them to the electronic tax return system.

[1467] Image processing technology: OpenCV is used to perform image recognition of document data.

[1468] Character recognition engine: Uses PyTesseract to extract characters in an image as text data.

[1469] Natural Language Processing (NLP): Uses technology to extract and organize information required for tax returns.

[1470] Emotion Engine: An artificial intelligence technology that analyzes a user's facial images and voice data to recognize their emotional state.

[1471] Specific example explanation

[1472] For example, consider the case where a user uses a receipt for hometown tax donations. The user takes a photo of the receipt using the camera on their smartphone. This document data is then uploaded to the server using the tax return assistant app. The server preprocesses the image using image processing technology (OpenCV) and extracts the text from the image using PyTesseract. It then uses natural language processing technology to identify necessary information such as the "amount" and "name of the local government" from the extracted text.

[1473] At the same time, the emotion engine analyzes the user's facial image and voice data to assess their emotional state (e.g., stress level). Based on this emotion data, the server provides the user with an optimal interface and displays necessary advice and help information.

[1474] Based on the analyzed data and emotional feedback, the information necessary for filing tax returns is generated and sent to the user's device. The user can then review the displayed information and make corrections or additional entries as necessary. During this process, the emotion engine monitors the user's emotional state and responds accordingly.

[1475] The server then automatically generates a tax return based on the final confirmed information and submits it to the electronic filing system. Once submission is complete, the server sends a notification of completion to the user via email or in-app notification.

[1476] Example prompts for generative AI models

[1477] "Please extract the amount and name of the local government from the hometown tax receipt, use an emotion engine to determine the user's stress level, and then automatically generate a tax return."

[1478] As described above, this system simplifies the complicated procedures users go through and provides optimal support according to their individual emotional state, thereby saving them a great deal of time and effort.

[1479] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1480] Program processing flow

[1481] Step 1:

[1482] The user takes a photo of the receipt or certificate required for tax return filing using a device such as a smartphone or PC. The captured image is temporarily saved in the device's local storage.

[1483] Specific behavior:

[1484] Input: receipt or certificate

[1485] Processing: Take a photo using the device's camera

[1486] Output: Image data saved in local storage

[1487] Step 2:

[1488] The user opens the Tax Return Assistant app and instructs it to upload the saved document data to the server. The device encrypts the image selected by the user and sends it to the server via a secure communication channel.

[1489] Specific behavior:

[1490] Input: Image data stored in local storage

[1491] Processing: Encryption of image data, transmission using HTTPS

[1492] Output: Image data uploaded to the server

[1493] Step 3:

[1494] The server analyzes the received image data using technologies such as OpenCV and PyTesseract, extracts character strings from the image, and saves them as text data.

[1495] Specific behavior:

[1496] Input: Image data uploaded to the server

[1497] Processing: Image preprocessing with OpenCV, character recognition with PyTesseract

[1498] Output: Text data

[1499] Step 4:

[1500] The server analyzes the acquired text data using natural language processing technology. The server identifies specific keywords and patterns, and extracts and organizes the necessary information (e.g., amount, date, name of local government).

[1501] Specific behavior:

[1502] Input: Text data

[1503] Processing: Applying NLP models to extract information

[1504] Output: Extracted information data (e.g. amount, date, name of local government)

[1505] Step 5:

[1506] The server or device uses an emotion engine to analyze the user's facial image and voice data to recognize the user's emotional state. The recognized emotional data is used throughout the system.

[1507] Specific behavior:

[1508] Input: User's facial image and voice data

[1509] Processing: Emotion recognition by emotion engine

[1510] Output: Emotion data

[1511] Step 6:

[1512] Based on the analyzed data and emotional feedback, the server generates optimal tax return information. The generated information is sent to the device, prompting the user to confirm and correct it. The emotion engine monitors the user's status throughout this process, providing help and advice as needed.

[1513] Specific behavior:

[1514] Input: Extracted information data, emotion data

[1515] Processing: Data formatting and generation

[1516] Output: Generated tax return information

[1517] Step 7:

[1518] The server automatically generates an official tax return based on the final verified information, which is then internally verified to ensure accuracy.

[1519] Specific behavior:

[1520] Input: Tax return information

[1521] Processing: Automatic generation and internal validation of tax returns

[1522] Output: Generated tax return

[1523] Step 8:

[1524] The server submits the verified tax return to the official electronic tax return system. Once the submission is complete, the server sends a notification to the user that the return has been completed. This notification can be sent via email or in-app notification.

[1525] Specific behavior:

[1526] Input: Generated tax return

[1527] Processing: Submission using the API of the electronic reporting system

[1528] Output: Notification of declaration completion

[1529] _Example of prompt sentence for generative AI model_

[1530] "Please extract the amount and name of the local government from the hometown tax receipt, use an emotion engine to determine the user's stress level, and then automatically generate a tax return."

[1531] (Application example 2)

[1532] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1533] In conventional tax return systems, users had to manually collect and analyze document data and generate tax returns, which was a cumbersome process that required time and effort. Furthermore, there was no mechanism to reduce the emotional burden caused by the stress and hassle, resulting in a poor user experience. Furthermore, there was no mechanism to generate and deliver interactive content based on the user's emotional state, making it difficult to personalize content based on the emotional state of each individual user.

[1534] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: [means for a user to read document data such as receipts and certificates;] [means for uploading the read data to the server;] [means for analyzing the data at the server and generating information necessary for filing a tax return;] [means for sending the generated information to the user's terminal so that the user can confirm and correct it;] [means for automatically generating a tax return based on the confirmed information and submitting it to an electronic filing system;] [means for optimizing the interface to the user based on the user's emotional state; and [means for generating story content based on the user's emotional data and delivering it interactively.] This simplifies the tax return process, reduces the emotional burden on the user, and enables personalized interactive content delivery based on the user's emotional state.

[1535] "User" refers to an individual who uses an application or system.

[1536] "Document data such as receipts and certificates" refers to digital data of official documents intended for the transfer of money or official certification.

[1537] "Server" refers to a computer system that stores, processes, and distributes data over a network.

[1538] "Analyzing data" refers to the process of processing acquired digital data and converting it into useful information.

[1539] "Information required for filing tax returns" refers to data such as amounts, dates, and personal information that are required when filing a tax return.

[1540] "Terminal" refers to a computer device such as a mobile phone, tablet, or PC that a user uses.

[1541] "Verify and correct" refers to the operation in which the user reviews the generated information and makes changes as necessary.

[1542] "Automatically generating tax returns" refers to the process by which the system automatically generates the necessary tax return documents based on the collected and analyzed data.

[1543] "Electronic filing system" refers to the official online platform for filing tax returns via the Internet.

[1544] "Emotional state" refers to data that indicates the user's mood or emotions at that time.

[1545] "Optimizing the interface" refers to improving the user experience by changing the system's display method and operating procedures according to the user's emotional state.

[1546] "Story content" refers to digital content in the form of a story that users can enjoy.

[1547] "Delivered interactively" means providing content in a format that allows users to progress through the story through choices.

[1548] The following description is given as a mode for carrying out the invention.

[1549] The present invention includes a tax return assistant system and an interactive story content distribution system. The system aims to enable individual users to complete tax return procedures easily and efficiently, and to provide personalized content based on the user's emotional state.

[1550] Hardware and Software

[1551] 1. Hardware

[1552] Devices: Smartphones, tablets, PCs, head-mounted displays, etc.

[1553] Server: A server capable of high-performance data processing

[1554] 2. Software

[1555] Image processing technology: OpenCV, PyTesseract

[1556] Natural language processing technology: NLTK, Transformers library

[1557] Emotion Recognition Engine: Emotion Recognition Model Using TensorFlow

[1558] Cloud storage: AWS S3

[1559] Web services: AWS Lambda, API Gateway

[1560] Details of data processing and calculation

[1561] 1. Obtaining and uploading document data

[1562] A user uses a device such as a smartphone to read document data such as receipts or certificates. The acquired data is temporarily stored in the device's local storage and then uploaded to a server via a dedicated application. This process uses the SSL / TLS protocol to maintain data confidentiality.

[1563] 2. Image Recognition and Text Data Extraction

[1564] The server uses OpenCV and PyTesseract to analyze uploaded document images and extract text data, such as the amount, date, and store name written on the receipt.

[1565] 3. Natural Language Processing of Text Data

[1566] The extracted text data is analyzed using NLTK and Transformers libraries. The server then accurately extracts and classifies the information required for tax returns. For example, it automatically assigns corresponding data to fields such as "amount," "date," and "address."

[1567] 4. Emotion Recognition and User Interface Optimization

[1568] An emotion recognition engine installed on the device or server analyzes the user's facial image and voice data to recognize their emotional state. This data is fed back into the interface design and help information display, helping to reduce the user's stress.

[1569] 5. Automatic generation and submission of tax return documents

[1570] The server automatically generates a tax return based on the data checked and corrected by the user. The generated return undergoes an internal validation process and is then submitted to the electronic filing system. After submission is complete, the server sends a completion notification to the user.

[1571] 6. Interactive story content generation and distribution

[1572] The server uses a generative AI model to generate story content based on the user's emotional state. Emotional data is used as prompts to dynamically generate story scenes. This content is delivered in an interactive format, allowing the user to progress through the story through choices.

[1573] Specific examples

[1574] For example, if a user is uploading image data for tax return and the process is stressful, the emotion recognition engine will detect this and provide a relaxing interface and friendly guidance for specific steps. After completing the tax return, the system will provide personalized story content to refresh the user.

[1575] Prompt Sentence Examples

[1576] Generate the following story scene based on the user's emotional data. The emotional state is excitement. The characters are a brave knight and a dragon. The location is a dark cave. The tone of the story is exciting and tense.

[1577] In this way, the system of the present invention can improve the user experience by simplifying the user's procedures and providing optimal support and entertainment tailored to their emotional state.

[1578] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1579] Step 1:

[1580] The user obtains document data such as receipts and certificates.

[1581] Specifically, the user takes an image of a document using a device such as a smartphone and temporarily stores it in the device's local storage.

[1582] Input: Document data (image format)

[1583] Output: Document data saved in local storage

[1584] Step 2:

[1585] The terminal uploads the document data to the server.

[1586] Specifically, the system uses a dedicated tax return assistant app to send the image selected by the user from the device to the server, while maintaining data confidentiality using the SSL / TLS protocol.

[1587] Input: Document data stored in local storage

[1588] Output: Document data uploaded to the server

[1589] Step 3:

[1590] The server analyzes the document data and performs image recognition.

[1591] Specifically, the server analyzes document images using OpenCV and PyTesseract and extracts text data.

[1592] Input: Document data uploaded to the server

[1593] Output: Extracted text data

[1594] Step 4:

[1595] The server analyzes the extracted text data using natural language processing.

[1596] Specifically, the server uses NLTK and Transformers libraries to analyze text data, extracting and classifying information necessary for tax returns. For example, it automatically identifies data corresponding to fields such as "amount," "date," and "address."

[1597] Input: Extracted text data

[1598] Output: Analyzed information (data required for tax return)

[1599] Step 5:

[1600] The server transmits the analyzed information to the user terminal.

[1601] Specifically, the information is sent to the user's device appropriately so that the user can check and modify it, and the interface is optimized to reduce user stress.

[1602] Input: Parsed information

[1603] Output: Information sent to the user's device

[1604] Step 6:

[1605] The terminal recognizes the user's emotional state.

[1606] Specifically, the system collects facial images and voice data using the device's camera and microphone, and determines the user's emotional state using an emotion recognition engine using TensorFlow.

[1607] Input: face image, voice data

[1608] Output: Emotional state data

[1609] Step 7:

[1610] A server generates story content based on emotional states.

[1611] Specifically, it uses emotion data as prompts to generate digital content (e.g., story scenes) using generative AI models, specifically generating exciting and suspenseful stories based on heightened emotional states.

[1612] Input: Emotional state data

[1613] Output: Generated story content

[1614] Step 8:

[1615] The server interactively delivers the generated story content to the user.

[1616] Specifically, the generated content is delivered to the user's device, allowing the user to progress through the story through choices. This delivery is done in real time, and the story changes dynamically depending on the user's choices.

[1617] Input: Generated story content

[1618] Output: Interactive story content delivered to user devices

[1619] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1620] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1621] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1622] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1623] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1624] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1625] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1626] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1627] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1628] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1629] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1630] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1631] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1632] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1633] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1634] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1635] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1636] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1637] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1638] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1639] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1640] The following is further disclosed regarding the above embodiment.

[1641] (Claim 1)

[1642] [Means for users to read document data such as receipts and certificates,

[1643] [Means for uploading the read data to a server;

[1644] [Means for analyzing data on the server and generating information necessary for tax returns;

[1645] [Means for transmitting the generated information to the user's terminal and for the user to confirm and modify it;

[1646] [Means for automatically generating tax returns based on the confirmed information and submitting them to an electronic tax return system;

[1647] A system including:

[1648] (Claim 2)

[1649] [means for using image processing techniques to perform image recognition of document data;

[1650] [Means for using natural language processing techniques to analyze text data extracted from the images; and

[1651] 10. The system of claim 1, comprising:

[1652] (Claim 3)

[1653] [Means for providing an interface for users to manually correct and add input;

[1654] [Method to automatically generate tax returns and verify them internally based on the confirmed data,

[1655] 10. The system of claim 1, comprising:

[1656] "Example 1"

[1657] (Claim 1)

[1658] [Means for users to read document data;

[1659] [Means for uploading the read data to a server;

[1660] [Means for analyzing data on the server and generating information necessary for tax returns;

[1661] [Means for transmitting the generated information to the user's terminal and for the user to confirm and modify it;

[1662] [Means for automatically generating tax returns based on the confirmed information and submitting them to an electronic tax return system;

[1663] A system including:

[1664] (Claim 2)

[1665] [means for using image processing techniques to perform image recognition of document data;

[1666] [Means for using natural language processing techniques to analyze text data extracted from the images; and

[1667] [Means using JSON format to structure data and

[1668] 10. The system of claim 1, comprising:

[1669] (Claim 3)

[1670] [Means for providing an interface for users to manually correct and add input;

[1671] [Method to automatically generate tax returns and verify them internally based on the confirmed data,

[1672] [Means for transmitting data to a server using a secure communications protocol; and

[1673] 10. The system of claim 1, comprising:

[1674] "Application Example 1"

[1675] (Claim 1)

[1676] [Means for users to read document data such as receipts and certificates,

[1677] [Means for uploading the read data to a server;

[1678] [Means for analyzing data on the server and generating information necessary for tax returns;

[1679] [Means for transmitting the generated information to the user's terminal and for the user to confirm and modify it;

[1680] [Means for automatically generating tax returns based on the confirmed information and submitting them to an electronic tax return system;

[1681] [Means for using an application installed on a smartphone, smart glasses, or head-mounted display to easily manage the read data in the store via electronic devices;

[1682] A system including:

[1683] (Claim 2)

[1684] [means for using image processing techniques to perform image recognition of document data;

[1685] [Means for using natural language processing techniques to analyze text data extracted from the images; and

[1686] The system of claim 1, further comprising: means for providing information generated based on the parsed data in JSON format.

[1687] (Claim 3)

[1688] [Means for providing an interface for users to manually correct and add input;

[1689] [Method to automatically generate tax returns and verify them internally based on the confirmed data,

[1690] [Means for inputting prompt sentences into a generative AI model using an electronic device to organize data acquired by a user at the time of purchase at a store for tax return purposes;

[1691] 10. The system of claim 1, comprising:

[1692] "Example 2: Combining Emotion Engines"

[1693] (Claim 1)

[1694] [Means for users to read document data such as receipts and certificates,

[1695] [Means for uploading the read data to a server;

[1696] [Means for analyzing data on the server and generating information necessary for tax returns;

[1697] [Means for transmitting the generated information to the user's terminal and for the user to confirm and modify it;

[1698] [Means for automatically generating tax returns based on the confirmed information and submitting them to an electronic tax return system;

[1699] [A means of capturing emotional states and feeding that data back into the system as a whole;

[1700] [Means for adjusting the interface based on the user's emotional state; and

[1701] A system including:

[1702] (Claim 2)

[1703] [means for using image processing techniques to perform image recognition of document data;

[1704] [Means for using natural language processing techniques to analyze text data extracted from the images; and

[1705] 10. The system of claim 1, comprising:

[1706] (Claim 3)

[1707] [Means for providing an interface for users to manually correct and add input;

[1708] [Method to automatically generate tax returns and verify them internally based on the confirmed data,

[1709] [Means for determining the user's stress state using an emotion engine and displaying appropriate help information and advice;

[1710] 10. The system of claim 1, comprising:

[1711] "Application example 2 when combining emotion engines"

[1712] (Claim 1)

[1713] [Means for users to read document data such as receipts and certificates,

[1714] [Means for uploading the read data to a server;

[1715] [Means for analyzing data on the server and generating information necessary for tax returns;

[1716] [Means for transmitting the generated information to the user's terminal and for the user to confirm and modify it;

[1717] [Means for automatically generating tax returns based on the confirmed information and submitting them to an electronic tax return system;

[1718] [Means for optimizing an interface to a user based on the user's emotional state;

[1719] [Means for generating story content based on user emotion data and delivering it interactively;

[1720] A system including:

[1721] (Claim 2)

[1722] [The system of claim 1, which uses image processing technology to perform image recognition of document data.

[1723] (Claim 3)

[1724] The system of claim 1, wherein the system uses natural language processing techniques to analyze the text data extracted from the image.

[1725] (Claim 4)

[1726] [The system of claim 1 provides an interface that allows a user to manually make corrections and additional inputs.

[1727] (Claim 5)

[1728] [The system according to claim 1, which automatically generates and internally verifies a tax return based on the confirmed data.

[1729] (Claim 6)

[1730] [The system according to claim 1, wherein the content of the story content is dynamically changed based on the emotional data acquired by the emotional engine.

[1731] (Claim 7)

[1732] [The system of claim 1, which uses a generative AI model to generate story scenes based on emotional states and present them to the user. [Explanation of symbols]

[1733] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for a user to read document data such as receipts and certificates; means for uploading the read data to a server; A means for analyzing the data on a server and generating information necessary for filing tax returns; A means for transmitting the generated information to a user's terminal so that the user can confirm and modify it; A means for automatically generating a tax return based on the confirmed information and submitting it to an electronic tax return system; A system including:

2. means for using image processing techniques to perform image recognition of the document data; a means for using natural language processing techniques to analyze the text data extracted from the image; The system of claim 1 , comprising:

3. means for providing an interface for users to manually correct and add information; Based on the confirmed data, a means to automatically generate a tax return and verify it internally, The system of claim 1 , comprising:

Images