System

Abstract

A system is provided.SOLUTION: A system comprising: means for capturing an image of a receipt; means for transmitting the captured image of the receipt to a server; optical character recognition means for extracting character data from the received image; means for classifying the extracted character data into specific categories; means for aggregating the classified data by month; and means for storing and providing the aggregated data to a user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's world, many receipts are generated to manage daily expenses and business-related expenditures. However, properly organizing and classifying these receipts and efficiently ascertaining the necessary information is a burden for many people. In particular, when filing tax returns, specific information such as medical expenses and donations must be organized, but the effort and errors involved can be a major problem. In addition, it is not easy to keep track of detailed increases and decreases in expenditures when managing daily household finances. To solve these issues, there is a need for a system that can easily organize receipts and automatically classify and aggregate them. [Means for solving the problem]

[0005] The present invention provides the following means to eliminate the complexity of receipt management and enable efficient tracking of expenses. First, a means for capturing images of receipts is provided, allowing users to easily obtain receipts as digital data. Next, a means for transmitting the captured images of receipts to a server is provided, enabling centralized management. Furthermore, optical character recognition (OCR) means is used to extract character data from the images received by the server, recognizing the receipt information as text data. Next, a means for classifying the extracted character data into specific categories (e.g., food expenses, utility expenses, medical expenses, etc.) is provided, clarifying each expense item. Furthermore, a means for aggregating the classified data by month is provided, automatically calculating monthly expenses. Additionally, a means for organizing and saving information required for filing tax returns (e.g., medical expenses, donations, etc.) is provided, reducing the cumbersome work of filing. Furthermore, a preprocessing means for correcting misrecognition of extracted character data is provided, improving data accuracy. These means provide a system that enables efficient receipt management and visualization of expenses.

[0006] The "means for capturing an image of a receipt" refers to a photographing device or software function that a user uses to obtain a receipt as digital data.

[0007] The "means for transmitting the photographed image of the receipt to the server" is a communication protocol or software function for securely transferring image data photographed on the terminal to the server.

[0008] "Optical character recognition means" is a software function that uses OCR (Optical Character Recognition) technology to extract character information from received image data.

[0009] The "means for classifying into specific categories" refers to an algorithm or software function that automatically classifies the extracted character data into categories such as food expenses, utility expenses, and medical expenses.

[0010] The "means for aggregating data by month" is a software function for summing up the data for each classified category on a monthly basis and organizing expenses.

[0011] "Means for storing and providing to users" refers to software functions for storing the organized and aggregated data in a digital format and providing it in a format that users can access.

[0012] "Means for organizing and storing information required for tax returns" refers to a software function that automatically extracts, organizes, and safely stores specific information required for tax returns, such as medical expenses and donations.

[0013] "Preprocessing means" refers to an algorithm or software function that corrects misrecognition of extracted character data and converts it into accurate data. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] A specific embodiment for implementing a receipt management system according to the present invention will now be described.

[0036] The system begins when a user takes a photo of a receipt using a device such as a smartphone or tablet and sends the image to a server. The server then analyzes the received image data and extracts the necessary information using optical character recognition (OCR) technology. The extracted text data is then classified by category, and monthly expenditures are calculated based on this. Finally, the aggregated data is saved in a format that can be viewed and checked by the user.

[0037] Program processing

[0038] Terminal side processing

[0039] 1. Launch the app

[0040] A user launches a receipt management app, which has a camera function and a server communication function.

[0041] 2. Take a photo of your receipt

[0042] The user takes a photo of the receipt using the app's camera function, and the image is saved in temporary storage on the device.

[0043] 3. Sending image data

[0044] The device encrypts the photo data and sends it to a server over the Internet, ensuring secure communication.

[0045] Server-side processing

[0046] 4. Receiving image data

[0047] The server receives and stores image data sent from the device, and the received data is automatically registered in the processing queue.

[0048] 5. OCR Processing

[0049] The server performs optical character recognition (OCR) on the image data to extract the text information written on the receipt, which is then saved as text data.

[0050] 6. Data Preprocessing

[0051] The server corrects misrecognition and formatting inconsistencies in character data extracted by OCR processing.

[0052] 7. Categorization

[0053] The server uses the preprocessed text data to classify expenses into categories such as food expenses, utility expenses, and medical expenses, using a machine learning algorithm.

[0054] 8. Monthly tally

[0055] The server aggregates the classified data by month and calculates the total expenditure amount and expenditure amount by category.

[0056] 9. Data Retention

[0057] The collected data is stored digitally so that users can check it later.

[0058] User behavior

[0059] 10. Data Verification

[0060] Users can check the aggregated spending data through the app, and can view monthly spending status and breakdowns by category.

[0061] 11. Receiving Advice

[0062] The user receives advice from the system. For example, if this month's food expenses have increased compared to last month, the app will suggest "reviewing your food expenses."

[0063] Specific examples

[0064] Example 1

[0065] A user goes shopping at a supermarket and receives a receipt.

[0066] The user launches the app and takes a photo of the receipt.

[0067] The device sends the captured image to the server.

[0068] The server analyzes the received image using OCR processing and extracts the text data.

[0069] The server classifies the extracted data as food expenses and reflects this in the monthly total amount.

[0070] Users can check their total food expenses through the app, which helps them manage their future spending.

[0071] Example 2

[0072] The user visits a hospital and receives a receipt.

[0073] The user uses the app to take a photo of the receipt.

[0074] The device sends the photo to the server.

[0075] The server analyzes the images and classifies them as medical expenses.

[0076] The server compiles medical expense data by month and organizes it as data for tax returns.

[0077] Users can check this month's medical expenses using the app, which can be useful when filing their tax returns.

[0078] In this way, the system of the present invention provides a means for efficiently managing receipts and visualizing expenditures.

[0079] The processing flow will be explained below.

[0080] Program processing

[0081] Step 1:

[0082] Subject: User

[0083] The user launches a receipt management app on their smartphone or tablet. The app has a photo capture function and a server communication function.

[0084] Step 2:

[0085] Subject: User

[0086] The user takes a photo of the receipt using the app's camera function, and the image is saved in the device's temporary storage.

[0087] Step 3:

[0088] Subject: Terminal

[0089] The device encrypts the captured image data, ensuring the security of the data.

[0090] Step 4:

[0091] Subject: Terminal

[0092] The device sends the encrypted image data to a server via the Internet using a secure communication protocol.

[0093] Step 5:

[0094] Subject: Server

[0095] The server receives the image data sent from the terminal and temporarily stores it. The received data is automatically added to the processing queue.

[0096] Step 6:

[0097] Subject: Server

[0098] The server performs optical character recognition (OCR) on the image data it receives, extracting the text information written on the receipt and saving it as text data.

[0099] Step 7:

[0100] Subject: Server

[0101] The server performs pre-processing to correct misrecognition and formatting inconsistencies in the character data extracted by OCR, thereby improving the accuracy of the data.

[0102] Step 8:

[0103] Subject: Server

[0104] The server classifies the preprocessed text data into specific categories (e.g., food expenses, utility bills, medical expenses) using machine learning algorithms.

[0105] Step 9:

[0106] Subject: Server

[0107] The server aggregates the categorized data by month and calculates the total expenditure for each category. The aggregated results are stored in a database and converted into a format that can be viewed by the user.

[0108] Step 10:

[0109] Subject: Server

[0110] The server organizes the results into the format required for tax returns and stores specific information such as medical expenses and donations separately.

[0111] Step 11:

[0112] Subject: User

[0113] Users can check the aggregated spending data through the app. Users can view monthly spending status and breakdowns by category.

[0114] Step 12:

[0115] Subject: User

[0116] The user receives advice from the system. For example, if food expenses have increased from last month, the user will receive a notification to "consider reviewing food expenses."

[0117] In this way, by performing specific processing for each step, the system efficiently manages receipts and makes expenditures visible.

[0118] Example 1

[0119] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0120] In recent years, receipt management has become increasingly complex, and there is a growing need to efficiently process, classify, and aggregate large volumes of receipts. Furthermore, to utilize receipt information for tax returns and expense management, it is essential to accurately and quickly extract data, correct misidentifications, and classify it into the necessary categories. Furthermore, to make it easier for users to manage their daily expenses, a system that allows easy data storage and viewing and provides appropriate advice is required.

[0121] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0122] In this invention, the server includes optical character recognition means for extracting character data from received images, means using a machine learning algorithm for classifying the extracted character data into specific categories, and preprocessing means for correcting misrecognition and formatting inconsistencies in the extracted character data. This allows for accurate extraction of text data from receipt images, classification into appropriate categories, and monthly aggregation. The system also includes means for organizing and saving information necessary for tax returns, thereby streamlining expense management and tax processing. Furthermore, a function for providing advice to users facilitates expense management.

[0123] The "means for taking a picture of a receipt" is a function that allows a user to take a picture of a receipt using a terminal.

[0124] The "means for transmitting the photographed image of the receipt to the server" is a function that enables the terminal to encrypt the photographed image data and transfer it to the server via the Internet.

[0125] "Optical character recognition means for extracting character data from received images" is a technology in which the server analyzes image data and extracts character information written on the receipt as text data.

[0126] "Means using a machine learning algorithm to classify extracted character data into specific categories" refers to an algorithm that the server uses machine learning technology to classify extracted character data into categories such as food expenses, utility expenses, and medical expenses.

[0127] The "means for aggregating classified data by month" is a function that allows the server to calculate the total monthly expenditure and the expenditure by category based on the data classified by category.

[0128] "Means for storing aggregated data and providing it to users" refers to the function that enables the server to safely store the aggregated data and allow users to view and check it later.

[0129] "Preprocessing means for correcting misrecognition or formatting inconsistencies in extracted character data" refers to a process in which the server corrects characters misrecognized during OCR processing or formatting inconsistencies to generate accurate text data.

[0130] The "means for providing advice to the user" is a function by which the server provides financial advice and suggestions for spending management based on the user's spending data.

[0131] A specific embodiment for implementing a receipt management system according to the present invention will be described. The purpose of this system is to efficiently manage receipts that users receive on a daily basis and to tally and classify expenditure data.

[0132] The system starts when a user takes a picture of a receipt using a device such as a smartphone or tablet. A receipt management app is installed on this device, which integrates a camera function and a server communication function. When a user launches the app and takes a picture of a receipt using the camera function, the image is saved in temporary storage on the device.

[0133] The device then encrypts the captured photo data and sends it over the Internet to a server using AES (Advanced Encryption Standard) encryption and HTTPS (Hyper Text Transfer Protocol Secure) to ensure secure communications.

[0134] The server receives the image data sent from the device and securely stores it in storage. The received data is automatically queued for processing. Next, the server analyzes the received image data using optical character recognition (OCR) technology to extract the text information listed on the receipt. Specifically, Google® Cloud Vision API or other OCR software is used. Because the text data extracted by OCR processing may contain misrecognition or formatting inconsistencies, the server performs preprocessing to correct these. This preprocessing includes spell checking and text formatting.

[0135] After preprocessing, the text data is classified into specific categories using machine learning algorithms. Specific algorithms used for this purpose include libraries such as scikit-learn and TENSORFLOW (registered trademark). The categories include food expenses, utility expenses, medical expenses, etc.

[0136] The classified data is aggregated monthly by the server, efficiently organized using an SQL database, and then securely stored in digital infrastructure such as Amazon S3 or Google Cloud Storage, where users can review it later.

[0137] Users can check the collected expenditure data through the app. When users press the "Check Data" button, monthly expenditure status and breakdown by category will be displayed. The system also has a function that allows users to receive financial advice. For example, if this month's food expenses have increased compared to last month, the system will suggest that users "review their food expenses."

[0138] Specific examples

[0139] Example 1

[0140] A user goes shopping at a supermarket and receives a receipt.

[0141] The user launches the app and takes a photo of the receipt.

[0142] The images captured by the device are encrypted using AES and sent to the server.

[0143] The server processes the images received using the Google Cloud Vision API and extracts text data.

[0144] The server classifies the extracted data as food expenses and reflects this in the monthly total amount.

[0145] The user checks the total food expenses through the app and receives the advice, "This month's food expenses have increased by 20% compared to last month."

[0146] Example 2

[0147] The user visits a hospital and receives a receipt.

[0148] The user uses the app to take a photo of the receipt.

[0149] The device encrypts the photo and sends it to the server.

[0150] The server analyzes the image using OCR processing and classifies it as medical expenses.

[0151] The server compiles medical expense data by month and organizes it as data for tax returns.

[0152] The user checks this month's medical expenses in the app and receives advice that "medical expense deductions may be applicable when filing your tax return."

[0153] Example prompts for generative AI models

[0154] "Describe in natural language the process of a receipt management system, from a user taking a photo of a receipt to a server parsing it to generate spending data. Explain each step of the process in detail and provide two concrete examples."

[0155] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0156] Step 1: Launch the app

[0157] A user launches a receipt management app on a smartphone or tablet. The app integrates a camera function and a server communication function. The input is the user's operation, and the output is the running status of the app.

[0158] process:

[0159] The user taps the app icon to launch it.

[0160] The app will launch and display the home screen.

[0161] Step 2: Take a photo of your receipt

[0162] The user takes a photo of the receipt using the camera function in the app. When the user presses the capture button, the camera takes a photo and the image is saved in temporary storage on the device. The input is the execution of the camera function, and the output is the captured image.

[0163] process:

[0164] The user presses the "Take Photo" button.

[0165] The device activates the camera, and the user frames the receipt and then presses the shutter button.

[0166] The image is saved to temporary storage.

[0167] Step 3: Sending image data

[0168] The device encrypts the captured photo data and sends it to a server over the Internet. This process uses AES encryption and the HTTPS protocol. The input is the image data, and the output is the encrypted data and its transmission.

[0169] process:

[0170] The device reads the image data from temporary storage.

[0171] Encrypt image data using AES.

[0172] The encrypted data is sent to the server using HTTPS.

[0173] Step 4: Receiving image data

[0174] The server receives image data sent from the device and stores it securely in storage. The received data is automatically registered in the processing queue. The input is encrypted image data, and the output is decrypted image data and its storage.

[0175] process:

[0176] The server receives the encrypted data using the HTTPS protocol.

[0177] The received data is decrypted using AES.

[0178] The decrypted data is stored in the server's storage and queued for processing.

[0179] Step 5: OCR processing

[0180] The server analyzes the received image data using optical character recognition (OCR) technology and extracts text information. Specifically, OCR software is used. The input is the decoded image data, and the output is the extracted text data.

[0181] process:

[0182] The server calls the Google Cloud Vision API or other OCR software to analyze the image data.

[0183] The text information written on the receipt is extracted using OCR processing.

[0184] The extracted text data is temporarily saved.

[0185] Step 6: Preprocessing the data

[0186] The server corrects misrecognition and formatting inconsistencies in the character data extracted by the OCR process. This process checks the spelling and formatting of the text data. The input is the extracted text data, and the output is the corrected text data.

[0187] process:

[0188] The server performs a spell check on the extracted text data.

[0189] Detect and correct formatting inconsistencies.

[0190] The corrected text data is saved.

[0191] Step 7: Categorize

[0192] The server classifies the preprocessed text data into categories using machine learning algorithms. The input is the corrected text data, and the output is the data classified into categories.

[0193] process:

[0194] The server invokes the machine learning model using libraries such as scikit-learn and TensorFlow.

[0195] The preprocessed data is fed into the model and categorical labels are assigned.

[0196] Store classified data.

[0197] Step 8: Monthly Tally

[0198] The server aggregates the classified data by month and calculates the total expenditure amount and expenditure amount by category. The input is data classified by category, and the output is the aggregated data.

[0199] process:

[0200] The server aggregates the categorized data by month using a database query.

[0201] Calculate total expenses and expenses by category for each month.

[0202] Save the summary results.

[0203] Step 9: Save your data

[0204] The server stores the aggregated data for users to review later. The input is the aggregated data, and the output is the stored data.

[0205] process:

[0206] The server stores the data on a digital platform such as Amazon S3 or Google Cloud Storage.

[0207] Verify that the saved data is stored correctly.

[0208] Step 10: Data verification

[0209] The user checks the aggregated spending data through the app. The input is the user request, and the output is the displayed spending data.

[0210] process:

[0211] The user presses the "Check Data" button in the app.

[0212] The server sends the stored aggregated data to the app in response to a user request.

[0213] The device displays the received data, allowing the user to check monthly spending and breakdowns by category.

[0214] Step 11: Getting advice

[0215] The user receives advice from the system: the input is expenditure data and the output is an advice message.

[0216] process:

[0217] The server analyzes spending data to detect unusual patterns and areas for improvement.

[0218] An advice message is generated and notified to the user.

[0219] Users can review advice messages to help them manage their future spending.

[0220] (Application example 1)

[0221] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0222] Traditional receipt management systems require the manual management of paper receipts, which is time-consuming and labor-intensive. They also lack the ability to efficiently import electronic receipts, making overall spending management cumbersome. They also make it difficult for users to understand their spending trends and identify areas for improvement. Furthermore, extracted character data can be misrecognized or incompletely classified, making accurate spending management difficult.

[0223] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0224] In this invention, the server includes a means for transmitting an image of a receipt to the server, an optical character recognition means for extracting character data from the received image, a means for classifying the extracted character data into specific categories, a means for automatically capturing electronic receipts and transmitting them to the system, and a means for providing the user with spending advice based on the compiled data. This allows users to efficiently manage not only paper receipts but also electronic receipts, making it easy to understand spending trends and areas for improvement. Furthermore, accurate classification of the extracted character data makes user spending management more accurate and efficient.

[0225] "Means for taking an image of a receipt" refers to a mobile device equipped with a camera function for capturing a digital image of a paper receipt.

[0226] "Means for transmitting the photographed image of the receipt to the server" refers to a communication function for transferring the digital image data of the acquired receipt to the server via a network.

[0227] "Optical character recognition means" means optical character recognition (OCR) technology for identifying characters from a digital image and extracting them as text data.

[0228] "Means for classifying extracted character data into specific categories" refers to algorithms or machine learning models that automatically organize and classify text data obtained through OCR into specific categories, such as food expenses and medical expenses.

[0229] "Means for aggregating classified data by month" refers to a program that aggregates data classified into specific categories on a monthly basis and calculates total expenditures and expenditures by category.

[0230] "Means for storing aggregated data and providing it to users" refers to the interface that stores monthly aggregated data in the server and allows users to check and view that data.

[0231] "Means for automatically receiving and transmitting electronic receipts to the system" refers to a function for automatically receiving electronic receipts generated from electronic payment services and managing them on the system.

[0232] "Means for providing spending advice to users based on aggregated data" refers to functionality that analyzes aggregated data and makes suggestions for improving or managing a user's spending habits.

[0233] This invention begins when a user takes a photo of a receipt using a device such as a smartphone and sends the data to a server. The server then analyzes the received image data using optical character recognition (OCR) technology, classifies the extracted text data by category, and aggregates it by month. This system allows users to efficiently manage their expenses. It also has a function that links with electronic payment services to automatically import electronic receipts and send them to the system.

[0234] Hardware and Software

[0235] Hardware:

[0236] Smartphone or tablet camera: Used to take a picture of the receipt.

[0237] Server: Used to receive, analyze, and store image data.

[0238] software:

[0239] OpenCV: Used for image preprocessing (grayscale conversion and binarization).

[0240] pytesseract: A library for OCR processing, used to extract text data from images.

[0241] scikit-learn: Used to classify character data extracted by machine learning algorithms into categories.

[0242] Program processing explanation

[0243] The server first receives the image data sent by the user and preprocesses the image using OpenCV. Next, it uses pytesseract to extract text data from the preprocessed image. It then uses the scikit-learn machine learning algorithm to classify the extracted text data into specific categories. The classified data is aggregated by month, and the total expenditure amount and expenditure amount by category are calculated. The results are stored on the server and made available for users to view and check through the application.

[0244] The company has also introduced a system for automatically capturing digital receipts, allowing users to manage both paper and digital receipts in one place, and it also has a function that provides users with spending advice based on the aggregated data.

[0245] Examples and prompts

[0246] Example 1

[0247] Let's say a user goes shopping at a supermarket and receives a receipt. When the user launches the smartphone app and takes a photo of the receipt, the device sends the image to the server. The server analyzes the received image using OCR processing and extracts text data. The system then classifies the extracted data as food expenses and reflects it in the monthly total. The user can check the total food expenses through the app, which can be used to manage future expenses.

[0248] Example 2

[0249] When a user receives a medical receipt from a hospital, they use the app to take a photo of the receipt, and the device sends the photo to the server. The server analyzes the image and classifies it as medical expenses. This data is aggregated monthly and organized as tax return data. Users can check their medical expenses for the month in the app, which can be useful when filing their tax return.

[0250] Prompt Sentence Examples

[0251] "A user's food spending this month has increased by 20% compared to last month. Please provide advice on how to address this increase."

[0252] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0253] Step 1:

[0254] The user launches the application and takes a picture of the receipt. When the user takes a picture of a paper receipt using a smartphone or tablet, the image is temporarily saved on the device. The input is the image of the receipt, and the output is the temporarily saved digital image data.

[0255] Step 2:

[0256] The device sends the captured image data to the server. The device transfers the encrypted image data to the server via the Internet. The input is the temporarily stored digital image data, and the output is the image data sent to the server.

[0257] Step 3:

[0258] The server stores the received image data and performs preprocessing for OCR. The server uses OpenCV to convert the image to grayscale and perform binarization. The input is the image data sent to the server, and the output is the preprocessed image data.

[0259] Step 4:

[0260] The server performs OCR processing on the preprocessed image and extracts character data. The server extracts the text data using pytesseract and saves it as character data in UTF-8 format. The input is the preprocessed image data, and the output is the extracted character data.

[0261] Step 5:

[0262] The server preprocesses the extracted character data to check for misrecognitions and inconsistencies. The server uses regular expressions to clean the extracted data and correct misrecognitions. The input is the extracted character data, and the output is the preprocessed, clean character data.

[0263] Step 6:

[0264] The server classifies the preprocessed text data into specific categories. The server uses scikit-learn's machine learning algorithm to automatically classify the data into categories such as food expenses and medical expenses. The input is the preprocessed, clean text data, and the output is the classified category data.

[0265] Step 7:

[0266] The server aggregates the classified data by month and calculates the total expenditure amount and expenditure amount by category. The server uses an SQL database to store and aggregate the data and organize it by month. The input is the classified category data, and the output is the expenditure data aggregated by month.

[0267] Step 8:

[0268] The server stores the aggregated data and provides it to the user. The server stores the aggregated results in a database and allows the user to view the data through the app. The input is monthly aggregated expenditure data, and the output is the stored aggregated data and the data displayed in the user interface.

[0269] Step 9:

[0270] The server provides spending advice to the user based on the aggregated data. The server uses a generative AI model to analyze the user's spending data and generate advice based on prompts. The input is monthly aggregated spending data and prompts, and the output is the generated advice.

[0271] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0272] A specific embodiment of the receipt management system combined with an emotion engine according to the present invention will now be described.

[0273] The system begins when a user takes a photo of a receipt using a device such as a smartphone or tablet and sends the image to a server. The server analyzes the received image data and extracts the necessary information using optical character recognition (OCR). The extracted text data is classified by category, and monthly expenditures are calculated based on this. The system then uses an emotion engine to recognize the user's emotions and customize spending advice based on this.

[0274] Program processing

[0275] Terminal side processing

[0276] 1. Launch the app

[0277] A user launches a receipt management app that integrates photo capture, emotion recognition, and server communication functions.

[0278] 2. Take a photo of your receipt

[0279] The user takes a photo of the receipt using the app's camera function, and the image is saved in the device's temporary storage.

[0280] 3. Sending image data

[0281] The device encrypts the photo data and sends it to a server over the Internet, ensuring secure communication.

[0282] Server-side processing

[0283] 4. Receiving image data

[0284] The server receives the image data sent from the terminal and temporarily stores it. The received data is automatically added to the processing queue.

[0285] 5. OCR Processing

[0286] The server performs optical character recognition (OCR) on the image data, extracting the text information written on the receipt and saving it as text data.

[0287] 6. Data Preprocessing

[0288] The server performs pre-processing to correct misrecognition and formatting inconsistencies in the character data extracted by OCR, thereby improving the accuracy of the data.

[0289] 7. Categorization

[0290] The server classifies the preprocessed text data into specific categories (e.g., food expenses, utility bills, medical expenses) using machine learning algorithms.

[0291] 8. Monthly tally

[0292] The server aggregates the categorized data by month and calculates the total expenditure for each category. The aggregated results are stored in a database and converted into a format that can be viewed by the user.

[0293] 9. Emotion recognition

[0294] The server uses an emotion engine to recognize the user's emotions from the user's input data and interaction logs, for example, by analyzing facial expressions and usage patterns displayed while the user is using the app.

[0295] 10. Customizing Advice

[0296] The server then customizes spending advice based on the perceived emotion, for example, gently suggesting savings advice if the user is feeling stressed.

[0297] 11. Data Retention

[0298] The aggregated data and customized advice will be stored digitally for users to review at a later date.

[0299] User behavior

[0300] 12. Data Verification

[0301] Users can check the aggregated spending data through the app. Users can view monthly spending status and breakdowns by category.

[0302] 13. Receiving Advice

[0303] The user receives customized advice from the system, for example, if their food expenses have increased since last month, the system will display advice tailored to their current emotional state.

[0304] Specific examples

[0305] Example 1

[0306] A user goes shopping at a supermarket and receives a receipt.

[0307] The user launches the app and takes a photo of the receipt.

[0308] The device sends the captured image to the server.

[0309] The server analyzes the received image using OCR processing and extracts the text data.

[0310] The server classifies the extracted data as food expenses and reflects this in the monthly total amount.

[0311] The server recognizes the user's emotions and makes gentle suggestions to save money if stress levels are high.

[0312] Users can check the total food cost and advice through the app to help manage their future spending.

[0313] Example 2

[0314] The user visits a hospital and receives a receipt.

[0315] The user uses the app to take a photo of the receipt.

[0316] The device sends the photo to the server.

[0317] The server analyzes the images and classifies them as medical expenses.

[0318] The server compiles medical expense data by month and organizes it as data for tax returns.

[0319] The server analyzes the user's emotions and, if anxiety is high, provides reassuring advice on reviewing spending, for example.

[0320] Users can check this month's medical expenses using the app, which can be useful when filing their tax returns.

[0321] In this way, the system of the present invention efficiently manages receipts and visualizes expenditures, and by combining it with an emotion engine, it provides expenditure advice tailored to the user's emotional state.

[0322] The processing flow will be explained below.

[0323] Program processing

[0324] Step 1:

[0325] Subject: User

[0326] A user launches a receipt management app on a smartphone or tablet, which is equipped with a camera, emotion recognition, and server communication functions.

[0327] Step 2:

[0328] Subject: User

[0329] The user takes a photo of the receipt using the app's camera function, and the image is saved in the device's temporary storage.

[0330] Step 3:

[0331] Subject: Terminal

[0332] The device encrypts the captured image data, ensuring the security of the data.

[0333] Step 4:

[0334] Subject: Terminal

[0335] The device sends the encrypted image data to a server via the Internet using a secure communication protocol.

[0336] Step 5:

[0337] Subject: Server

[0338] The server receives the image data sent from the terminal and temporarily stores it. The received data is automatically added to the processing queue.

[0339] Step 6:

[0340] Subject: Server

[0341] The server performs optical character recognition (OCR) on the image data it receives, extracting the text information written on the receipt and saving it as text data.

[0342] Step 7:

[0343] Subject: Server

[0344] The server performs pre-processing to correct misrecognition and formatting inconsistencies in the character data extracted by OCR, thereby improving the accuracy of the data.

[0345] Step 8:

[0346] Subject: Server

[0347] The server classifies the preprocessed text data into specific categories (e.g., food expenses, utility bills, medical expenses) using machine learning algorithms.

[0348] Step 9:

[0349] Subject: Server

[0350] The server aggregates the categorized data by month and calculates the total expenditure for each category. The aggregated results are stored in a database and converted into a format that can be viewed by the user.

[0351] Step 10:

[0352] Subject: Server

[0353] The server uses an emotion engine to recognize the user's emotions from the user's input data and interaction logs, for example, by analyzing facial expressions and usage patterns displayed while the user is using the app.

[0354] Step 11:

[0355] Subject: Server

[0356] The server then customizes spending advice based on the perceived emotion, for example, gently suggesting savings advice if the user is feeling stressed.

[0357] Step 12:

[0358] Subject: Server

[0359] The server then stores the results and customized advice digitally, allowing users to review them later.

[0360] Step 13:

[0361] Subject: User

[0362] Users can check the aggregated spending data through the app. Users can view monthly spending status and breakdowns by category.

[0363] Step 14:

[0364] Subject: User

[0365] The user receives customized advice from the system, for example, if their food expenses have increased since last month, the system will display advice tailored to their current emotional state.

[0366] Example 2

[0367] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0368] In modern society, managing and organizing the large number of receipts generated in daily life and business situations is a significant burden for users. Taking images of receipts, extracting information, and accurately categorizing and aggregating them is particularly time-consuming and labor-intensive. Furthermore, since appropriate spending advice is not provided based on the user's emotional state, it can easily lead to stressful situations. Conventional technologies have not been able to adequately resolve these issues.

[0369] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0370] In this invention, the server includes means for capturing an image of a receipt, means for transmitting the captured image of the receipt to the server, optical character recognition means for extracting character data from the received image, means for classifying the extracted character data into specific categories, means for aggregating the classified data by month, means for recognizing a user's emotion, means for customizing advice on spending based on the recognized emotion, and means for saving the aggregated data and customized advice and providing it to the user. This not only improves the efficiency of receipt management work but also enables spending advice tailored to the user's emotional state.

[0371] A "receipt" is a document issued to prove the content and amount of a transaction when purchasing a product or using a service.

[0372] An "image" is visual information recorded in digital form, and is data that includes text, graphics, etc.

[0373] A "server" is a computer system that provides data and services to other devices (terminals) within a network.

[0374] A "terminal" is a communication device or computer that can only function when connected to a network.

[0375] Optical character recognition (OCR) is a technology that reads character information in an image and converts it into digital text.

[0376] A "category" is a division for classifying data according to its type or characteristics.

[0377] "Aggregation" is the process of aggregating multiple data sets and calculating statistical information such as totals and averages.

[0378] "Emotion recognition" is a technology that analyzes and determines a user's emotional state from data.

[0379] "Advice" is advice or suggestions offered to encourage optimal action or decision in a particular situation.

[0380] "Storage" is the act of writing down data on a recording medium for later use.

[0381] "User" refers to an individual or corporation that uses this system to manage receipts and check data.

[0382] This section describes a specific example of combining an emotion engine with a receipt management system according to the present invention. This system begins when a user takes an image of a receipt using a device such as a smartphone or tablet and sends the image to a server. The specific hardware used is a smartphone or tablet, and the software used is a dedicated receipt management app.

[0383] The server-side processing includes receiving image data, OCR processing, data preprocessing, categorization, monthly aggregation, emotion recognition, and generating and providing customized advice. These processes are performed using the following technologies:

[0384] Server-side processing

[0385] 1. Receiving image data

[0386] The server receives the image data sent from the device and temporarily stores it. The HTTPS protocol is used for communication, and the AES encryption algorithm ensures data security.

[0387] 2. OCR processing

[0388] The server uses optical character recognition technology, such as Google Cloud Vision API, to extract text from the image data. This process converts the text data on the receipt into text format.

[0389] 3. Data Preprocessing

[0390] Character data extracted by OCR may contain misrecognition errors or formatting inconsistencies, and the server uses a Python-based data cleaning tool to correct these errors.

[0391] 4. Categorization

[0392] A machine learning algorithm (e.g., the BERT model) is used to classify the preprocessed text data into specific categories (e.g., food expenses, utility expenses, medical expenses), thereby achieving accurate data classification.

[0393] 5. Monthly tally

[0394] The categorized data is aggregated by month, and the total expenditure for each category is calculated. The aggregated results are stored in a database (e.g., MySQL (registered trademark)) and can be viewed by users in HTML or JSON format.

[0395] 6. Emotion recognition

[0396] We use the Microsoft® Azure® Emotion API to recognize the user's emotional state from input data and interaction logs, allowing us to accurately determine the user's emotional state (e.g., stress, relief).

[0397] 7. Customizing Advice

[0398] Based on the recognized emotions, spending advice is customized using a generative AI model (e.g., OpenAI's GPT-4®). For example, it generates "food expenses are increasing, but reasonable savings suggestions are being generated due to high stress."

[0399] 8. Data storage and provision

[0400] The aggregated data and customized advice is stored on Amazon S3 or Google Cloud Storage, which users can then review later through the app.

[0401] User behavior

[0402] Users can view aggregated spending data through the app. They can view monthly spending status and category breakdowns in graphs and tables for specific spending management. They can also receive customized spending advice, allowing them to review their spending appropriately based on their emotional state.

[0403] Specific examples

[0404] As a concrete example, let's consider the process of a user shopping at a supermarket, receiving a receipt, and then taking a photo of the receipt using the app. The device sends this photo data to a server, which then analyzes the image using OCR to extract text data. The extracted data is classified as "food expenses" and reflected in the monthly total. If the user's stress level is high using emotion recognition, gentle suggestions for saving money are made, which the user can confirm through the app and use to help manage future expenses.

[0405] Next, when the user visits a hospital for a medical examination and receives a receipt, the same procedure is followed. The app takes a photo of the receipt, and the server classifies it as a medical expense. The medical expense data is aggregated by month and organized for tax return filing. Furthermore, if the user is highly anxious based on sentiment analysis, the app provides advice on reviewing expenses to reassure them.

[0406] Example prompt using a generative AI model: "My food expenses have been increasing recently, and I'm under a lot of stress. Can you tell me some reasonable ways to save money?"

[0407] The above is a specific embodiment for implementing the present invention. This system efficiently manages receipts and visualizes expenditures, and by combining it with an emotion engine, it provides expenditure advice tailored to the user's emotional state.

[0408] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0409] Step 1:

[0410] Launching the app

[0411] The user launches the receipt management app on their smartphone or tablet. The app prompts the user for authentication information (e.g., ID and password) the first time it is launched. If authentication is successful, the user is redirected to the main screen.

[0412] Input: Launching an app through user interaction

[0413] Output: The main screen of the app is displayed.

[0414] Step 2:

[0415] Taking a photo of the receipt

[0416] The user takes a photo of the receipt using the app's camera, and the image is saved in temporary device storage in JPEG format, ready to be sent to the server later.

[0417] Input: User-taken image of receipt

[0418] Output: JPEG image data is saved to temporary storage

[0419] Step 3:

[0420] Sending image data

[0421] The device encrypts the captured image data using the 128-bit AES encryption algorithm, then sends the encrypted image data to the server using HTTPS, with the TLS protocol ensuring communication security.

[0422] Input: JPEG format image data

[0423] Output: The encrypted image data is sent to the server.

[0424] Step 4:

[0425] Receiving image data

[0426] The server receives the image data sent from the device and temporarily stores it. The received data is automatically added to the processing queue and awaits analysis.

[0427] Input: Encrypted image data

[0428] Output: Temporarily saved image data

[0429] Step 5:

[0430] OCR processing

[0431] The server uses the Google Cloud Vision API to perform OCR processing on the image data, extracting the text information from the receipt and saving it as text data.

[0432] Input: Temporarily saved image data

[0433] Output: Extracted text data

[0434] Step 6:

[0435] Data Preprocessing

[0436] The server performs pre-processing to correct misrecognition and formatting inconsistencies in the character data extracted by OCR. For example, if the number "1" is misrecognized as a "lowercase l" or an "uppercase I," it will be converted to the correct format.

[0437] Input: Extracted text data

[0438] Output: Preprocessed text data

[0439] Step 7:

[0440] Categorization

[0441] The server classifies the preprocessed text data into specific categories (e.g., food expenses, utility expenses, medical expenses) using a machine learning algorithm (e.g., the BERT model).

[0442] Input: Preprocessed text data

[0443] Output: Categorized data

[0444] Step 8:

[0445] Monthly tally

[0446] The server aggregates the categorized data by month and calculates the total expenditure for each category. The aggregated results are stored in a database (e.g., MySQL) and can be viewed by users in HTML or JSON format.

[0447] Input: Categorical data

[0448] Output: Monthly aggregated data

[0449] Step 9:

[0450] emotion recognition

[0451] The server uses Microsoft Azure's Emotion API to recognize the user's emotional state from the user's input data and interaction logs, thereby accurately determining the user's emotional state (e.g., stress, relief).

[0452] Input: User input data and interaction logs

[0453] Output: Recognized emotion data

[0454] Step 10:

[0455] Customizing Advice

[0456] The server then uses a generative AI model (e.g., OpenAI's GPT-4) to customize spending advice based on the recognized emotions. For example, it generates a reasonable saving suggestion, saying, "Food expenses are increasing, but stress is high."

[0457] Input: Recognized emotion data

[0458] Output: Customized advice

[0459] Step 11:

[0460] Data storage and provision

[0461] The server stores the aggregated data and customized advice on Amazon S3 or Google Cloud Storage, which users can then view via the app.

[0462] Input: Monthly aggregated data and customized advice

[0463] Output: Stored data and advice

[0464] Step 12:

[0465] Data confirmation

[0466] The user accesses the "Data Check" section of the app, where they can view their monthly spending and a breakdown by category in graphs and tables.

[0467] Input: Data confirmation request by user operation

[0468] Output: Screen showing expenditure status and breakdown

[0469] Step 13:

[0470] Receiving advice

[0471] The user accesses the "Advice" section and sees customized spending advice from the system. For example, because food expenses have increased since last month, soft suggestions are displayed that are tailored to the user's emotional state.

[0472] Input: User initiated advice request

[0473] Output: Screen showing customized advice

[0474] (Application example 2)

[0475] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0476] In today's brick-and-mortar shopping environment, customers lack the means to efficiently manage receipts and understand their spending in real time. Furthermore, when customers experience emotional stress related to their spending habits, they often lack the ability to receive appropriate advice and support. There is a need for a system that can solve these problems and provide more personalized spending management and psychological support to customers.

[0477] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0478] In this invention, the server includes means for capturing an image of a receipt, means for transmitting the captured image of the receipt to the server, optical character recognition means for extracting character data from the received image, means for classifying the extracted character data into specific categories, means for aggregating the classified data by month, means for storing and providing the aggregated data to a user, and means for recognizing the user's emotions and customizing spending advice based on the emotion data, thereby enabling customers to manage their spending in real time while shopping at a physical store and receive optimal spending advice based on their emotions.

[0479] The "means for taking a picture of a receipt" is a function that allows a user to take a picture of a receipt for a product purchased at a physical store using the camera on a mobile device such as a smartphone.

[0480] The "means for transmitting the photographed image of the receipt to the server" is a communication function that encrypts the photographed image of the receipt and transmits it securely to the server.

[0481] "Optical character recognition means for extracting character data from received images" refers to a function that uses optical character recognition (OCR) technology to analyze character information from receipt images received by the server and extract it as text data.

[0482] The "means for classifying extracted character data into specific categories" is a function for classifying character data extracted by OCR into specific categories such as food, daily necessities, and medical expenses.

[0483] The "means for aggregating classified data by month" is a function for aggregating expenditure data classified by category by month.

[0484] The "means for storing aggregated data and providing it to the user" is a function for storing the aggregated results in a database and displaying them so that the user can check them through the app.

[0485] "Means for recognizing user emotions and customizing spending advice based on this emotional data" refers to a function that analyzes user interactions and input data to recognize emotions and provides individually optimized spending advice based on the results of this recognition.

[0486] This invention is a system that allows customers in physical stores to use their smartphones to manage receipts and visualize spending in real time, and provides personalized advice based on their emotions. Specific embodiments of the invention are described below.

[0487] System Overview

[0488] It consists of three elements: a server, a terminal (such as a smartphone), and a user. Details of each element are as follows:

[0489] Required Hardware and Software

[0490] Hardware: Smartphone (compatible with ANDROID (registered trademark) and iOS), store Wi-Fi network, cloud server

[0491] Software: smartphone applications, OCR libraries (e.g., Tesseract), emotion recognition algorithms (e.g., Microsoft Azure Cognitive Services or Google Cloud Vision API)

[0492] Processing flow and data processing

[0493] 1. Terminal processing

[0494] App launch: A user enters a store and launches the app, which has integrated photo-taking and emotion recognition functions.

[0495] Taking a photo of a receipt: The user uses the camera function within the app to take a photo of the receipt. The captured image is saved in the device's temporary storage.

[0496] Transmission of image data: The device encrypts the image data and transmits it to a server via the Internet.

[0497] 2. Server-side processing

[0498] Receiving image data: The server receives the image data sent from the terminal and temporarily stores it.

[0499] OCR processing: The server performs OCR processing on the image data received and extracts the text information written on the receipt.

[0500] Data preprocessing: Preprocessing is performed to correct misrecognition and formatting inconsistencies in the character data extracted by OCR processing.

[0501] Category Classification: Classify the preprocessed text data into specific categories such as food, daily necessities, medical expenses, etc., using machine learning algorithms (scikit-learn, TensorFlow).

[0502] Monthly aggregation: The classified data is aggregated by month to calculate the total expenditure amount for each category.

[0503] Emotion Recognition: Analyzes user interactions and input data and recognizes user emotions with the emotion engine. Emotion recognition is performed using Microsoft Azure Cognitive Services or Google Cloud Vision API.

[0504] Customized Advice: Customize spending advice based on recognized emotions. Customized advice is generated using recommendation algorithms.

[0505] 3. User Actions

[0506] Data confirmation: Users can check the aggregated expenditure data through the app. Users can view monthly expenditure status and breakdown by category.

[0507] Get advice: Users receive personalized spending advice based on their emotions.

[0508] Specific examples

[0509] Example 1

[0510] The user goes shopping at the grocery store and takes a photo of the receipt they receive at the register. The app uses OCR to classify it as a "food expense" and, after analyzing their past purchase history and emotional data, displays advice such as, "Your food expenses have increased compared to last month, so why not take advantage of the weekend sales?"

[0511] Example prompts for generative AI models

[0512] After a user shops at a physical store, they take a photo of the receipt with the app. The app then performs OCR processing to analyze the receipt's contents. The OCR-extracted text data is then categorized and aggregated by month. The app also analyzes the user's emotions and provides personalized spending advice based on their emotions.

[0513] In this way, the system of the present invention enhances the shopping experience in brick-and-mortar stores and gives customers peace of mind in managing their spending.

[0514] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0515] Step 1:

[0516] Launching the app

[0517] A user enters a physical store and launches the app on their smartphone, which has integrated photo-taking and emotion recognition functions.

[0518] Input: None

[0519] Output: App launch

[0520] Step 2:

[0521] Taking a photo of the receipt

[0522] A user purchases an item at a store and takes a photo of the receipt they receive at the cash register using the app's camera function.

[0523] Input: Receipt

[0524] Output: Receipt image data

[0525] Step 3:

[0526] Sending image data

[0527] The device encrypts the image of the receipt it captures and sends it to a cloud server via the Internet.

[0528] Input: Receipt image data

[0529] Output: Image data sent to the server

[0530] Step 4:

[0531] Receiving image data

[0532] The server receives the image data sent from the terminal and temporarily stores it.

[0533] Input: Image data sent

[0534] Output: Image data saved in temporary storage

[0535] Step 5:

[0536] OCR processing

[0537] The server performs OCR processing on the image data received and extracts the text information written on the receipt. An OCR library such as Tesseract is used.

[0538] Input: Image data

[0539] Output: Extracted character data

[0540] Step 6:

[0541] Data Preprocessing

[0542] The server corrects misrecognition and formatting inconsistencies in the text data extracted by the OCR process, for example, correcting misrecognized letters and numbers and arranging them into a consistent format.

[0543] Input: Extracted character data

[0544] Output: Preprocessed character data

[0545] Step 7:

[0546] Categorization

[0547] The server classifies the preprocessed text data into specific categories such as food, daily necessities, medical expenses, etc. It uses machine learning algorithms such as scikit-learn and TensorFlow.

[0548] Input: Preprocessed character data

[0549] Output: Categorized data

[0550] Step 8:

[0551] Monthly tally

[0552] The server aggregates the classified data by month and performs data calculations to calculate the total expenditure amount for each category.

[0553] Input: Categorical data

[0554] Output: Monthly aggregated data

[0555] Step 9:

[0556] emotion recognition

[0557] The server analyzes the user's interactions and input data while using the app, and performs data calculations to recognize the user's emotions using an emotion engine (such as Microsoft Azure Cognitive Services or Google Cloud Vision API). It analyzes the user's facial expressions and vocabulary patterns.

[0558] Input: Interaction data and input data

[0559] Output: Emotion recognition result

[0560] Step 10:

[0561] Customizing Advice

[0562] The server generates customized spending advice based on the perceived emotions, using recommendation algorithms such as collaborative filtering.

[0563] Input: Emotion recognition results and monthly aggregated data

[0564] Output: Customized spending advice

[0565] Step 11:

[0566] Data confirmation

[0567] Users can access aggregated spending data and personalized advice through the app. Users can view monthly spending and a breakdown by category.

[0568] Input: Monthly aggregated data and customized advice

[0569] Output: Spending data and advice displayed in a user interface

[0570] In this way, the system helps users manage their spending through a series of steps and provides optimal emotional advice.

[0571] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0572] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0573] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0574] [Second embodiment]

[0575] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0576] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0577] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0578] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0579] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0580] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0581] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0582] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0583] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0584] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0585] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0586] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0587] A specific embodiment for implementing a receipt management system according to the present invention will now be described.

[0588] The system begins when a user takes a photo of a receipt using a device such as a smartphone or tablet and sends the image to a server. The server then analyzes the received image data and extracts the necessary information using optical character recognition (OCR) technology. The extracted text data is then classified by category, and monthly expenditures are calculated based on this. Finally, the aggregated data is saved in a format that can be viewed and checked by the user.

[0589] Program processing

[0590] Terminal side processing

[0591] 1. Launch the app

[0592] A user launches a receipt management app, which has a camera function and a server communication function.

[0593] 2. Take a photo of your receipt

[0594] The user takes a photo of the receipt using the app's camera function, and the image is saved in temporary storage on the device.

[0595] 3. Sending image data

[0596] The device encrypts the photo data and sends it to a server over the Internet, ensuring secure communication.

[0597] Server-side processing

[0598] 4. Receiving image data

[0599] The server receives and stores image data sent from the device, and the received data is automatically registered in the processing queue.

[0600] 5. OCR Processing

[0601] The server performs optical character recognition (OCR) on the image data to extract the text information written on the receipt, which is then saved as text data.

[0602] 6. Data Preprocessing

[0603] The server corrects misrecognition and formatting inconsistencies in character data extracted by OCR processing.

[0604] 7. Categorization

[0605] The server uses the preprocessed text data to classify expenses into categories such as food expenses, utility expenses, and medical expenses, using a machine learning algorithm.

[0606] 8. Monthly tally

[0607] The server aggregates the classified data by month and calculates the total expenditure amount and expenditure amount by category.

[0608] 9. Data Retention

[0609] The collected data is stored digitally so that users can check it later.

[0610] User behavior

[0611] 10. Data Verification

[0612] Users can check the aggregated spending data through the app, and can view monthly spending status and breakdowns by category.

[0613] 11. Receiving Advice

[0614] The user receives advice from the system. For example, if this month's food expenses have increased compared to last month, the app will suggest "reviewing your food expenses."

[0615] Specific examples

[0616] Example 1

[0617] A user goes shopping at a supermarket and receives a receipt.

[0618] The user launches the app and takes a photo of the receipt.

[0619] The device sends the captured image to the server.

[0620] The server analyzes the received image using OCR processing and extracts the text data.

[0621] The server classifies the extracted data as food expenses and reflects this in the monthly total amount.

[0622] Users can check their total food expenses through the app, which helps them manage their future spending.

[0623] Example 2

[0624] The user visits a hospital and receives a receipt.

[0625] The user uses the app to take a photo of the receipt.

[0626] The device sends the photo to the server.

[0627] The server analyzes the images and classifies them as medical expenses.

[0628] The server compiles medical expense data by month and organizes it as data for tax returns.

[0629] Users can check this month's medical expenses using the app, which can be useful when filing their tax returns.

[0630] In this way, the system of the present invention provides a means for efficiently managing receipts and visualizing expenditures.

[0631] The processing flow will be explained below.

[0632] Program processing

[0633] Step 1:

[0634] Subject: User

[0635] The user launches a receipt management app on their smartphone or tablet. The app has a photo capture function and a server communication function.

[0636] Step 2:

[0637] Subject: User

[0638] The user takes a photo of the receipt using the app's camera function, and the image is saved in the device's temporary storage.

[0639] Step 3:

[0640] Subject: Terminal

[0641] The device encrypts the captured image data, ensuring the security of the data.

[0642] Step 4:

[0643] Subject: Terminal

[0644] The device sends the encrypted image data to a server via the Internet using a secure communication protocol.

[0645] Step 5:

[0646] Subject: Server

[0647] The server receives the image data sent from the terminal and temporarily stores it. The received data is automatically added to the processing queue.

[0648] Step 6:

[0649] Subject: Server

[0650] The server performs optical character recognition (OCR) on the image data it receives, extracting the text information written on the receipt and saving it as text data.

[0651] Step 7:

[0652] Subject: Server

[0653] The server performs pre-processing to correct misrecognition and formatting inconsistencies in the character data extracted by OCR, thereby improving the accuracy of the data.

[0654] Step 8:

[0655] Subject: Server

[0656] The server classifies the preprocessed text data into specific categories (e.g., food expenses, utility bills, medical expenses) using machine learning algorithms.

[0657] Step 9:

[0658] Subject: Server

[0659] The server aggregates the categorized data by month and calculates the total expenditure for each category. The aggregated results are stored in a database and converted into a format that can be viewed by the user.

[0660] Step 10:

[0661] Subject: Server

[0662] The server organizes the results into the format required for tax returns and stores specific information such as medical expenses and donations separately.

[0663] Step 11:

[0664] Subject: User

[0665] Users can check the aggregated spending data through the app. Users can view monthly spending status and breakdowns by category.

[0666] Step 12:

[0667] Subject: User

[0668] The user receives advice from the system. For example, if food expenses have increased from last month, the user will receive a notification to "consider reviewing food expenses."

[0669] In this way, by performing specific processing for each step, the system efficiently manages receipts and makes expenditures visible.

[0670] Example 1

[0671] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0672] In recent years, receipt management has become increasingly complex, and there is a growing need to efficiently process, classify, and aggregate large volumes of receipts. Furthermore, to utilize receipt information for tax returns and expense management, it is essential to accurately and quickly extract data, correct misidentifications, and classify it into the necessary categories. Furthermore, to make it easier for users to manage their daily expenses, a system that allows easy data storage and viewing and provides appropriate advice is required.

[0673] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0674] In this invention, the server includes optical character recognition means for extracting character data from received images, means using a machine learning algorithm for classifying the extracted character data into specific categories, and preprocessing means for correcting misrecognition and formatting inconsistencies in the extracted character data. This allows for accurate extraction of text data from receipt images, classification into appropriate categories, and monthly aggregation. The system also includes means for organizing and saving information necessary for tax returns, thereby streamlining expense management and tax processing. Furthermore, a function for providing advice to users facilitates expense management.

[0675] The "means for taking a picture of a receipt" is a function that allows a user to take a picture of a receipt using a terminal.

[0676] The "means for transmitting the photographed image of the receipt to the server" is a function that enables the terminal to encrypt the photographed image data and transfer it to the server via the Internet.

[0677] "Optical character recognition means for extracting character data from received images" is a technology in which the server analyzes image data and extracts character information written on the receipt as text data.

[0678] "Means using a machine learning algorithm to classify extracted character data into specific categories" refers to an algorithm that the server uses machine learning technology to classify extracted character data into categories such as food expenses, utility expenses, and medical expenses.

[0679] The "means for aggregating classified data by month" is a function that allows the server to calculate the total monthly expenditure and the expenditure by category based on the data classified by category.

[0680] "Means for storing aggregated data and providing it to users" refers to the function that enables the server to safely store the aggregated data and allow users to view and check it later.

[0681] "Preprocessing means for correcting misrecognition or formatting inconsistencies in extracted character data" refers to a process in which the server corrects characters misrecognized during OCR processing or formatting inconsistencies to generate accurate text data.

[0682] The "means for providing advice to the user" is a function by which the server provides financial advice and suggestions for spending management based on the user's spending data.

[0683] A specific embodiment for implementing a receipt management system according to the present invention will be described. The purpose of this system is to efficiently manage receipts that users receive on a daily basis and to tally and classify expenditure data.

[0684] The system starts when a user takes a picture of a receipt using a device such as a smartphone or tablet. A receipt management app is installed on this device, which integrates a camera function and a server communication function. When a user launches the app and takes a picture of a receipt using the camera function, the image is saved in temporary storage on the device.

[0685] The device then encrypts the captured photo data and sends it over the Internet to a server using AES (Advanced Encryption Standard) encryption and HTTPS (Hyper Text Transfer Protocol Secure) to ensure secure communications.

[0686] The server receives the image data sent from the device and securely stores it in storage. The received data is automatically queued for processing. The server then analyzes the received image data using optical character recognition (OCR) technology to extract the text information listed on the receipt. Specifically, Google Cloud Vision API or other OCR software is used. The text data extracted by the OCR process may contain recognition errors or formatting inconsistencies, so the server performs preprocessing to correct these. This preprocessing includes spell checking and text formatting.

[0687] After preprocessing, the text data is then classified into specific categories using machine learning algorithms, such as libraries like scikit-learn and TensorFlow, including categories like food, utility, and medical expenses.

[0688] The classified data is aggregated monthly by the server, efficiently organized using an SQL database, and then securely stored in digital infrastructure such as Amazon S3 or Google Cloud Storage, where users can review it later.

[0689] Users can check the collected expenditure data through the app. When users press the "Check Data" button, monthly expenditure status and breakdown by category will be displayed. The system also has a function that allows users to receive financial advice. For example, if this month's food expenses have increased compared to last month, the system will suggest that users "review their food expenses."

[0690] Specific examples

[0691] Example 1

[0692] A user goes shopping at a supermarket and receives a receipt.

[0693] The user launches the app and takes a photo of the receipt.

[0694] The images captured by the device are encrypted using AES and sent to the server.

[0695] The server processes the images received using the Google Cloud Vision API and extracts text data.

[0696] The server classifies the extracted data as food expenses and reflects this in the monthly total amount.

[0697] The user checks the total food expenses through the app and receives the advice, "This month's food expenses have increased by 20% compared to last month."

[0698] Example 2

[0699] The user visits a hospital and receives a receipt.

[0700] The user uses the app to take a photo of the receipt.

[0701] The device encrypts the photo and sends it to the server.

[0702] The server analyzes the image using OCR processing and classifies it as medical expenses.

[0703] The server compiles medical expense data by month and organizes it as data for tax returns.

[0704] The user checks this month's medical expenses in the app and receives advice that "medical expense deductions may be applicable when filing your tax return."

[0705] Example prompts for generative AI models

[0706] "Describe in natural language the process of a receipt management system, from a user taking a photo of a receipt to a server parsing it to generate spending data. Explain each step of the process in detail and provide two concrete examples."

[0707] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0708] Step 1: Launch the app

[0709] A user launches a receipt management app on a smartphone or tablet. The app integrates a camera function and a server communication function. The input is the user's operation, and the output is the running status of the app.

[0710] process:

[0711] The user taps the app icon to launch it.

[0712] The app will launch and display the home screen.

[0713] Step 2: Take a photo of your receipt

[0714] The user takes a photo of the receipt using the camera function in the app. When the user presses the capture button, the camera takes a photo and the image is saved in temporary storage on the device. The input is the execution of the camera function, and the output is the captured image.

[0715] process:

[0716] The user presses the "Take Photo" button.

[0717] The device activates the camera, and the user frames the receipt and then presses the shutter button.

[0718] The image is saved to temporary storage.

[0719] Step 3: Sending image data

[0720] The device encrypts the captured photo data and sends it to a server over the Internet. This process uses AES encryption and the HTTPS protocol. The input is the image data, and the output is the encrypted data and its transmission.

[0721] process:

[0722] The device reads the image data from temporary storage.

[0723] Encrypt image data using AES.

[0724] The encrypted data is sent to the server using HTTPS.

[0725] Step 4: Receiving image data

[0726] The server receives image data sent from the device and stores it securely in storage. The received data is automatically registered in the processing queue. The input is encrypted image data, and the output is decrypted image data and its storage.

[0727] process:

[0728] The server receives the encrypted data using the HTTPS protocol.

[0729] The received data is decrypted using AES.

[0730] The decrypted data is stored in the server's storage and queued for processing.

[0731] Step 5: OCR processing

[0732] The server analyzes the received image data using optical character recognition (OCR) technology and extracts text information. Specifically, OCR software is used. The input is the decoded image data, and the output is the extracted text data.

[0733] process:

[0734] The server calls the Google Cloud Vision API or other OCR software to analyze the image data.

[0735] The text information written on the receipt is extracted using OCR processing.

[0736] The extracted text data is temporarily saved.

[0737] Step 6: Preprocessing the data

[0738] The server corrects misrecognition and formatting inconsistencies in the character data extracted by the OCR process. This process checks the spelling and formatting of the text data. The input is the extracted text data, and the output is the corrected text data.

[0739] process:

[0740] The server performs a spell check on the extracted text data.

[0741] Detect and correct formatting inconsistencies.

[0742] The corrected text data is saved.

[0743] Step 7: Categorize

[0744] The server classifies the preprocessed text data into categories using machine learning algorithms. The input is the corrected text data, and the output is the data classified into categories.

[0745] process:

[0746] The server invokes the machine learning model using libraries such as scikit-learn and TensorFlow.

[0747] The preprocessed data is fed into the model and categorical labels are assigned.

[0748] Store classified data.

[0749] Step 8: Monthly Tally

[0750] The server aggregates the classified data by month and calculates the total expenditure amount and expenditure amount by category. The input is data classified by category, and the output is the aggregated data.

[0751] process:

[0752] The server aggregates the categorized data by month using a database query.

[0753] Calculate total expenses and expenses by category for each month.

[0754] Save the summary results.

[0755] Step 9: Save your data

[0756] The server stores the aggregated data for users to review later. The input is the aggregated data, and the output is the stored data.

[0757] process:

[0758] The server stores the data on a digital platform such as Amazon S3 or Google Cloud Storage.

[0759] Verify that the saved data is stored correctly.

[0760] Step 10: Data verification

[0761] The user checks the aggregated spending data through the app. The input is the user request, and the output is the displayed spending data.

[0762] process:

[0763] The user presses the "Check Data" button in the app.

[0764] The server sends the stored aggregated data to the app in response to a user request.

[0765] The device displays the received data, allowing the user to check monthly spending and breakdowns by category.

[0766] Step 11: Getting advice

[0767] The user receives advice from the system: the input is expenditure data and the output is an advice message.

[0768] process:

[0769] The server analyzes spending data to detect unusual patterns and areas for improvement.

[0770] An advice message is generated and notified to the user.

[0771] Users can review advice messages to help them manage their future spending.

[0772] (Application example 1)

[0773] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0774] Traditional receipt management systems require the manual management of paper receipts, which is time-consuming and labor-intensive. They also lack the ability to efficiently import electronic receipts, making overall spending management cumbersome. They also make it difficult for users to understand their spending trends and identify areas for improvement. Furthermore, extracted character data can be misrecognized or incompletely classified, making accurate spending management difficult.

[0775] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0776] In this invention, the server includes a means for transmitting an image of a receipt to the server, an optical character recognition means for extracting character data from the received image, a means for classifying the extracted character data into specific categories, a means for automatically capturing electronic receipts and transmitting them to the system, and a means for providing the user with spending advice based on the compiled data. This allows users to efficiently manage not only paper receipts but also electronic receipts, making it easy to understand spending trends and areas for improvement. Furthermore, accurate classification of the extracted character data makes user spending management more accurate and efficient.

[0777] "Means for taking an image of a receipt" refers to a mobile device equipped with a camera function for capturing a digital image of a paper receipt.

[0778] "Means for transmitting the photographed image of the receipt to the server" refers to a communication function for transferring the digital image data of the acquired receipt to the server via a network.

[0779] "Optical character recognition means" means optical character recognition (OCR) technology for identifying characters from a digital image and extracting them as text data.

[0780] "Means for classifying extracted character data into specific categories" refers to algorithms or machine learning models that automatically organize and classify text data obtained through OCR into specific categories, such as food expenses and medical expenses.

[0781] "Means for aggregating classified data by month" refers to a program that aggregates data classified into specific categories on a monthly basis and calculates total expenditures and expenditures by category.

[0782] "Means for storing aggregated data and providing it to users" refers to the interface that stores monthly aggregated data in the server and allows users to check and view that data.

[0783] "Means for automatically receiving and transmitting electronic receipts to the system" refers to a function for automatically receiving electronic receipts generated from electronic payment services and managing them on the system.

[0784] "Means for providing spending advice to users based on aggregated data" refers to functionality that analyzes aggregated data and makes suggestions for improving or managing a user's spending habits.

[0785] This invention begins when a user takes a photo of a receipt using a device such as a smartphone and sends the data to a server. The server then analyzes the received image data using optical character recognition (OCR) technology, classifies the extracted text data by category, and aggregates it by month. This system allows users to efficiently manage their expenses. It also has a function that links with electronic payment services to automatically import electronic receipts and send them to the system.

[0786] Hardware and Software

[0787] Hardware:

[0788] Smartphone or tablet camera: Used to take a picture of the receipt.

[0789] Server: Used to receive, analyze, and store image data.

[0790] software:

[0791] OpenCV: Used for image preprocessing (grayscale conversion and binarization).

[0792] pytesseract: A library for OCR processing, used to extract text data from images.

[0793] scikit-learn: Used to classify character data extracted by machine learning algorithms into categories.

[0794] Program processing explanation

[0795] The server first receives the image data sent by the user and preprocesses the image using OpenCV. Next, it uses pytesseract to extract text data from the preprocessed image. It then uses the scikit-learn machine learning algorithm to classify the extracted text data into specific categories. The classified data is aggregated by month, and the total expenditure amount and expenditure amount by category are calculated. The results are stored on the server and made available for users to view and check through the application.

[0796] The company has also introduced a system for automatically capturing digital receipts, allowing users to manage both paper and digital receipts in one place, and it also has a function that provides users with spending advice based on the aggregated data.

[0797] Examples and prompts

[0798] Example 1

[0799] Let's say a user goes shopping at a supermarket and receives a receipt. When the user launches the smartphone app and takes a photo of the receipt, the device sends the image to the server. The server analyzes the received image using OCR processing and extracts text data. The system then classifies the extracted data as food expenses and reflects it in the monthly total. The user can check the total food expenses through the app, which can be used to manage future expenses.

[0800] Example 2

[0801] When a user receives a medical receipt from a hospital, they use the app to take a photo of the receipt, and the device sends the photo to the server. The server analyzes the image and classifies it as medical expenses. This data is aggregated monthly and organized as tax return data. Users can check their medical expenses for the month in the app, which can be useful when filing their tax return.

[0802] Prompt Sentence Examples

[0803] "A user's food spending this month has increased by 20% compared to last month. Please provide advice on how to address this increase."

[0804] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0805] Step 1:

[0806] The user launches the application and takes a picture of the receipt. When the user takes a picture of a paper receipt using a smartphone or tablet, the image is temporarily saved on the device. The input is the image of the receipt, and the output is the temporarily saved digital image data.

[0807] Step 2:

[0808] The device sends the captured image data to the server. The device transfers the encrypted image data to the server via the Internet. The input is the temporarily stored digital image data, and the output is the image data sent to the server.

[0809] Step 3:

[0810] The server stores the received image data and performs preprocessing for OCR. The server uses OpenCV to convert the image to grayscale and perform binarization. The input is the image data sent to the server, and the output is the preprocessed image data.

[0811] Step 4:

[0812] The server performs OCR processing on the preprocessed image and extracts character data. The server extracts the text data using pytesseract and saves it as character data in UTF-8 format. The input is the preprocessed image data, and the output is the extracted character data.

[0813] Step 5:

[0814] The server preprocesses the extracted character data to check for misrecognitions and inconsistencies. The server uses regular expressions to clean the extracted data and correct misrecognitions. The input is the extracted character data, and the output is the preprocessed, clean character data.

[0815] Step 6:

[0816] The server classifies the preprocessed text data into specific categories. The server uses scikit-learn's machine learning algorithm to automatically classify the data into categories such as food expenses and medical expenses. The input is the preprocessed, clean text data, and the output is the classified category data.

[0817] Step 7:

[0818] The server aggregates the classified data by month and calculates the total expenditure amount and expenditure amount by category. The server uses an SQL database to store and aggregate the data and organize it by month. The input is the classified category data, and the output is the expenditure data aggregated by month.

[0819] Step 8:

[0820] The server stores the aggregated data and provides it to the user. The server stores the aggregated results in a database and allows the user to view the data through the app. The input is monthly aggregated expenditure data, and the output is the stored aggregated data and the data displayed in the user interface.

[0821] Step 9:

[0822] The server provides spending advice to the user based on the aggregated data. The server uses a generative AI model to analyze the user's spending data and generate advice based on prompts. The input is monthly aggregated spending data and prompts, and the output is the generated advice.

[0823] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0824] A specific embodiment of the receipt management system combined with an emotion engine according to the present invention will now be described.

[0825] The system begins when a user takes a photo of a receipt using a device such as a smartphone or tablet and sends the image to a server. The server analyzes the received image data and extracts the necessary information using optical character recognition (OCR). The extracted text data is classified by category, and monthly expenditures are calculated based on this. The system then uses an emotion engine to recognize the user's emotions and customize spending advice based on this.

[0826] Program processing

[0827] Terminal side processing

[0828] 1. Launch the app

[0829] A user launches a receipt management app that integrates photo capture, emotion recognition, and server communication functions.

[0830] 2. Take a photo of your receipt

[0831] The user takes a photo of the receipt using the app's camera function, and the image is saved in the device's temporary storage.

[0832] 3. Sending image data

[0833] The device encrypts the photo data and sends it to a server over the Internet, ensuring secure communication.

[0834] Server-side processing

[0835] 4. Receiving image data

[0836] The server receives the image data sent from the terminal and temporarily stores it. The received data is automatically added to the processing queue.

[0837] 5. OCR Processing

[0838] The server performs optical character recognition (OCR) on the image data, extracting the text information written on the receipt and saving it as text data.

[0839] 6. Data Preprocessing

[0840] The server performs pre-processing to correct misrecognition and formatting inconsistencies in the character data extracted by OCR, thereby improving the accuracy of the data.

[0841] 7. Categorization

[0842] The server classifies the preprocessed text data into specific categories (e.g., food expenses, utility bills, medical expenses) using machine learning algorithms.

[0843] 8. Monthly tally

[0844] The server aggregates the categorized data by month and calculates the total expenditure for each category. The aggregated results are stored in a database and converted into a format that can be viewed by the user.

[0845] 9. Emotion recognition

[0846] The server uses an emotion engine to recognize the user's emotions from the user's input data and interaction logs, for example, by analyzing facial expressions and usage patterns displayed while the user is using the app.

[0847] 10. Customizing Advice

[0848] The server then customizes spending advice based on the perceived emotion, for example, gently suggesting savings advice if the user is feeling stressed.

[0849] 11. Data Retention

[0850] The aggregated data and customized advice will be stored digitally for users to review at a later date.

[0851] User behavior

[0852] 12. Data Verification

[0853] Users can check the aggregated spending data through the app. Users can view monthly spending status and breakdowns by category.

[0854] 13. Receiving Advice

[0855] The user receives customized advice from the system, for example, if their food expenses have increased since last month, the system will display advice tailored to their current emotional state.

[0856] Specific examples

[0857] Example 1

[0858] A user goes shopping at a supermarket and receives a receipt.

[0859] The user launches the app and takes a photo of the receipt.

[0860] The device sends the captured image to the server.

[0861] The server analyzes the received image using OCR processing and extracts the text data.

[0862] The server classifies the extracted data as food expenses and reflects this in the monthly total amount.

[0863] The server recognizes the user's emotions and makes gentle suggestions to save money if stress levels are high.

[0864] Users can check the total food cost and advice through the app to help manage their future spending.

[0865] Example 2

[0866] The user visits a hospital and receives a receipt.

[0867] The user uses the app to take a photo of the receipt.

[0868] The device sends the photo to the server.

[0869] The server analyzes the images and classifies them as medical expenses.

[0870] The server compiles medical expense data by month and organizes it as data for tax returns.

[0871] The server analyzes the user's emotions and, if anxiety is high, provides reassuring advice on reviewing spending, for example.

[0872] Users can check this month's medical expenses using the app, which can be useful when filing their tax returns.

[0873] In this way, the system of the present invention efficiently manages receipts and visualizes expenditures, and by combining it with an emotion engine, it provides expenditure advice tailored to the user's emotional state.

[0874] The processing flow will be explained below.

[0875] Program processing

[0876] Step 1:

[0877] Subject: User

[0878] A user launches a receipt management app on a smartphone or tablet, which is equipped with a camera, emotion recognition, and server communication functions.

[0879] Step 2:

[0880] Subject: User

[0881] The user takes a photo of the receipt using the app's camera function, and the image is saved in the device's temporary storage.

[0882] Step 3:

[0883] Subject: Terminal

[0884] The device encrypts the captured image data, ensuring the security of the data.

[0885] Step 4:

[0886] Subject: Terminal

[0887] The device sends the encrypted image data to a server via the Internet using a secure communication protocol.

[0888] Step 5:

[0889] Subject: Server

[0890] The server receives the image data sent from the terminal and temporarily stores it. The received data is automatically added to the processing queue.

[0891] Step 6:

[0892] Subject: Server

[0893] The server performs optical character recognition (OCR) on the image data it receives, extracting the text information written on the receipt and saving it as text data.

[0894] Step 7:

[0895] Subject: Server

[0896] The server performs pre-processing to correct misrecognition and formatting inconsistencies in the character data extracted by OCR, thereby improving the accuracy of the data.

[0897] Step 8:

[0898] Subject: Server

[0899] The server classifies the preprocessed text data into specific categories (e.g., food expenses, utility bills, medical expenses) using machine learning algorithms.

[0900] Step 9:

[0901] Subject: Server

[0902] The server aggregates the categorized data by month and calculates the total expenditure for each category. The aggregated results are stored in a database and converted into a format that can be viewed by the user.

[0903] Step 10:

[0904] Subject: Server

[0905] The server uses an emotion engine to recognize the user's emotions from the user's input data and interaction logs, for example, by analyzing facial expressions and usage patterns displayed while the user is using the app.

[0906] Step 11:

[0907] Subject: Server

[0908] The server then customizes spending advice based on the perceived emotion, for example, gently suggesting savings advice if the user is feeling stressed.

[0909] Step 12:

[0910] Subject: Server

[0911] The server then stores the results and customized advice digitally, allowing users to review them later.

[0912] Step 13:

[0913] Subject: User

[0914] Users can check the aggregated spending data through the app. Users can view monthly spending status and breakdowns by category.

[0915] Step 14:

[0916] Subject: User

[0917] The user receives customized advice from the system, for example, if their food expenses have increased since last month, the system will display advice tailored to their current emotional state.

[0918] Example 2

[0919] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0920] In modern society, managing and organizing the large number of receipts generated in daily life and business situations is a significant burden for users. Taking images of receipts, extracting information, and accurately categorizing and aggregating them is particularly time-consuming and labor-intensive. Furthermore, since appropriate spending advice is not provided based on the user's emotional state, it can easily lead to stressful situations. Conventional technologies have not been able to adequately resolve these issues.

[0921] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0922] In this invention, the server includes means for capturing an image of a receipt, means for transmitting the captured image of the receipt to the server, optical character recognition means for extracting character data from the received image, means for classifying the extracted character data into specific categories, means for aggregating the classified data by month, means for recognizing a user's emotion, means for customizing advice on spending based on the recognized emotion, and means for saving the aggregated data and customized advice and providing it to the user. This not only improves the efficiency of receipt management work but also enables spending advice tailored to the user's emotional state.

[0923] A "receipt" is a document issued to prove the content and amount of a transaction when purchasing a product or using a service.

[0924] An "image" is visual information recorded in digital form, and is data that includes text, graphics, etc.

[0925] A "server" is a computer system that provides data and services to other devices (terminals) within a network.

[0926] A "terminal" is a communication device or computer that can only function when connected to a network.

[0927] Optical character recognition (OCR) is a technology that reads character information in an image and converts it into digital text.

[0928] A "category" is a division for classifying data according to its type or characteristics.

[0929] "Aggregation" is the process of aggregating multiple data sets and calculating statistical information such as totals and averages.

[0930] "Emotion recognition" is a technology that analyzes and determines a user's emotional state from data.

[0931] "Advice" is advice or suggestions offered to encourage optimal action or decision in a particular situation.

[0932] "Storage" is the act of writing down data on a recording medium for later use.

[0933] "User" refers to an individual or corporation that uses this system to manage receipts and check data.

[0934] This section describes a specific example of combining an emotion engine with a receipt management system according to the present invention. This system begins when a user takes an image of a receipt using a device such as a smartphone or tablet and sends the image to a server. The specific hardware used is a smartphone or tablet, and the software used is a dedicated receipt management app.

[0935] The server-side processing includes receiving image data, OCR processing, data preprocessing, categorization, monthly aggregation, emotion recognition, and generating and providing customized advice. These processes are performed using the following technologies:

[0936] Server-side processing

[0937] 1. Receiving image data

[0938] The server receives the image data sent from the device and temporarily stores it. The HTTPS protocol is used for communication, and the AES encryption algorithm ensures data security.

[0939] 2. OCR processing

[0940] The server uses optical character recognition technology, such as Google Cloud Vision API, to extract text from the image data. This process converts the text data on the receipt into text format.

[0941] 3. Data Preprocessing

[0942] The character data extracted by OCR may contain misrecognition errors or formatting inconsistencies, and the server uses a Python-based data cleaning tool to correct these errors.

[0943] 4. Categorization

[0944] A machine learning algorithm (e.g., the BERT model) is used to classify the preprocessed text data into specific categories (e.g., food expenses, utility expenses, medical expenses), thereby achieving accurate data classification.

[0945] 5. Monthly tally

[0946] The categorized data is aggregated by month, and the total expenditure for each category is calculated. The aggregated results are stored in a database (e.g., MySQL) and can be viewed by users in HTML or JSON format.

[0947] 6. Emotion recognition

[0948] We use Microsoft Azure's Emotion API to recognize the user's emotional state from the user's input data and interaction logs, which allows us to accurately determine the user's emotional state (e.g., stress, relief).

[0949] 7. Customizing Advice

[0950] Based on the recognized emotions, spending advice is customized using a generative AI model (e.g., OpenAI's GPT-4). For example, it generates "food expenses are increasing, but reasonable savings suggestions are generated due to high stress."

[0951] 8. Data storage and provision

[0952] The aggregated data and customized advice is stored on Amazon S3 or Google Cloud Storage, which users can then review later through the app.

[0953] User behavior

[0954] Users can view aggregated spending data through the app. They can view monthly spending status and category breakdowns in graphs and tables for specific spending management. They can also receive customized spending advice, allowing them to review their spending appropriately based on their emotional state.

[0955] Specific examples

[0956] As a concrete example, let's consider the process of a user shopping at a supermarket, receiving a receipt, and then taking a photo of the receipt using the app. The device sends this photo data to a server, which then analyzes the image using OCR to extract text data. The extracted data is classified as "food expenses" and reflected in the monthly total. If the user's stress level is high using emotion recognition, gentle suggestions for saving money are made, which the user can confirm through the app and use to help manage future expenses.

[0957] Next, when the user visits a hospital for a medical examination and receives a receipt, the same procedure is followed. The app takes a photo of the receipt, and the server classifies it as a medical expense. The medical expense data is aggregated by month and organized for tax return filing. Furthermore, if the user is highly anxious based on sentiment analysis, the app provides advice on reviewing expenses to reassure them.

[0958] Example prompt using a generative AI model: "My food expenses have been increasing recently, and I'm under a lot of stress. Can you tell me some reasonable ways to save money?"

[0959] The above is a specific embodiment for implementing the present invention. This system efficiently manages receipts and visualizes expenditures, and by combining it with an emotion engine, it provides expenditure advice tailored to the user's emotional state.

[0960] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0961] Step 1:

[0962] Launching the app

[0963] The user launches the receipt management app on their smartphone or tablet. The app prompts the user for authentication information (e.g., ID and password) the first time it is launched. If authentication is successful, the user is redirected to the main screen.

[0964] Input: Launching an app through user interaction

[0965] Output: The main screen of the app is displayed.

[0966] Step 2:

[0967] Taking a photo of the receipt

[0968] The user takes a photo of the receipt using the app's camera, and the image is saved in temporary device storage in JPEG format, ready to be sent to the server later.

[0969] Input: User-taken image of receipt

[0970] Output: JPEG image data is saved to temporary storage

[0971] Step 3:

[0972] Sending image data

[0973] The device encrypts the captured image data using the 128-bit AES encryption algorithm, then sends the encrypted image data to the server using HTTPS, with the TLS protocol ensuring communication security.

[0974] Input: JPEG format image data

[0975] Output: The encrypted image data is sent to the server.

[0976] Step 4:

[0977] Receiving image data

[0978] The server receives the image data sent from the device and temporarily stores it. The received data is automatically added to the processing queue and awaits analysis.

[0979] Input: Encrypted image data

[0980] Output: Temporarily saved image data

[0981] Step 5:

[0982] OCR processing

[0983] The server uses the Google Cloud Vision API to perform OCR processing on the image data, extracting the text information from the receipt and saving it as text data.

[0984] Input: Temporarily saved image data

[0985] Output: Extracted text data

[0986] Step 6:

[0987] Data Preprocessing

[0988] The server performs pre-processing to correct misrecognition and formatting inconsistencies in the character data extracted by OCR. For example, if the number "1" is misrecognized as a "lowercase l" or an "uppercase I," it will be converted to the correct format.

[0989] Input: Extracted text data

[0990] Output: Preprocessed text data

[0991] Step 7:

[0992] Categorization

[0993] The server classifies the preprocessed text data into specific categories (e.g., food expenses, utility expenses, medical expenses) using a machine learning algorithm (e.g., the BERT model).

[0994] Input: Preprocessed text data

[0995] Output: Categorized data

[0996] Step 8:

[0997] Monthly tally

[0998] The server aggregates the categorized data by month and calculates the total expenditure for each category. The aggregated results are stored in a database (e.g., MySQL) and can be viewed by users in HTML or JSON format.

[0999] Input: Categorical data

[1000] Output: Monthly aggregated data

[1001] Step 9:

[1002] emotion recognition

[1003] The server uses Microsoft Azure's Emotion API to recognize the user's emotional state from the user's input data and interaction logs, thereby accurately determining the user's emotional state (e.g., stress, relief).

[1004] Input: User input data and interaction logs

[1005] Output: Recognized emotion data

[1006] Step 10:

[1007] Customizing Advice

[1008] The server then uses a generative AI model (e.g., OpenAI's GPT-4) to customize spending advice based on the recognized emotions. For example, it generates a reasonable saving suggestion, saying, "Food expenses are increasing, but stress is high."

[1009] Input: Recognized emotion data

[1010] Output: Customized advice

[1011] Step 11:

[1012] Data storage and provision

[1013] The server stores the aggregated data and customized advice on Amazon S3 or Google Cloud Storage, which users can then view via the app.

[1014] Input: Monthly aggregated data and customized advice

[1015] Output: Stored data and advice

[1016] Step 12:

[1017] Data confirmation

[1018] The user accesses the "Data Check" section of the app, where they can view their monthly spending and a breakdown by category in graphs and tables.

[1019] Input: Data confirmation request by user operation

[1020] Output: Screen showing expenditure status and breakdown

[1021] Step 13:

[1022] Receiving advice

[1023] The user accesses the "Advice" section and sees customized spending advice from the system. For example, because food expenses have increased since last month, soft suggestions are displayed that are tailored to the user's emotional state.

[1024] Input: User initiated advice request

[1025] Output: Screen showing customized advice

[1026] (Application example 2)

[1027] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1028] In today's brick-and-mortar shopping environment, customers lack the means to efficiently manage receipts and understand their spending in real time. Furthermore, when customers experience emotional stress related to their spending habits, they often lack the ability to receive appropriate advice and support. There is a need for a system that can solve these problems and provide more personalized spending management and psychological support to customers.

[1029] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1030] In this invention, the server includes means for capturing an image of a receipt, means for transmitting the captured image of the receipt to the server, optical character recognition means for extracting character data from the received image, means for classifying the extracted character data into specific categories, means for aggregating the classified data by month, means for storing and providing the aggregated data to a user, and means for recognizing the user's emotions and customizing spending advice based on the emotion data, thereby enabling customers to manage their spending in real time while shopping at a physical store and receive optimal spending advice based on their emotions.

[1031] The "means for taking a picture of a receipt" is a function that allows a user to take a picture of a receipt for a product purchased at a physical store using the camera on a mobile device such as a smartphone.

[1032] The "means for transmitting the photographed image of the receipt to the server" is a communication function that encrypts the photographed image of the receipt and transmits it securely to the server.

[1033] "Optical character recognition means for extracting character data from received images" refers to a function that uses optical character recognition (OCR) technology to analyze character information from receipt images received by the server and extract it as text data.

[1034] The "means for classifying extracted character data into specific categories" is a function for classifying character data extracted by OCR into specific categories such as food, daily necessities, and medical expenses.

[1035] The "means for aggregating classified data by month" is a function for aggregating expenditure data classified by category by month.

[1036] The "means for storing aggregated data and providing it to the user" is a function for storing the aggregated results in a database and displaying them so that the user can check them through the app.

[1037] "Means for recognizing user emotions and customizing spending advice based on this emotional data" refers to a function that analyzes user interactions and input data to recognize emotions and provides individually optimized spending advice based on the results of this recognition.

[1038] This invention is a system that allows customers in physical stores to use their smartphones to manage receipts and visualize spending in real time, and provides personalized advice based on their emotions. Specific embodiments of the invention are described below.

[1039] System Overview

[1040] It consists of three elements: a server, a terminal (such as a smartphone), and a user. Details of each element are as follows:

[1041] Required Hardware and Software

[1042] Hardware: Smartphone (compatible with Android and iOS), store Wi-Fi network, cloud server

[1043] Software: smartphone applications, OCR libraries (e.g., Tesseract), emotion recognition algorithms (e.g., Microsoft Azure Cognitive Services or Google Cloud Vision API)

[1044] Processing flow and data processing

[1045] 1. Terminal processing

[1046] App launch: A user enters a store and launches the app, which has integrated photo-taking and emotion recognition functions.

[1047] Taking a photo of a receipt: The user uses the camera function within the app to take a photo of the receipt. The captured image is saved in the device's temporary storage.

[1048] Transmission of image data: The device encrypts the image data and transmits it to a server via the Internet.

[1049] 2. Server-side processing

[1050] Receiving image data: The server receives the image data sent from the terminal and temporarily stores it.

[1051] OCR processing: The server performs OCR processing on the image data received and extracts the text information written on the receipt.

[1052] Data preprocessing: Preprocessing is performed to correct misrecognition and formatting inconsistencies in the character data extracted by OCR processing.

[1053] Category Classification: Classify the preprocessed text data into specific categories such as food, daily necessities, medical expenses, etc., using machine learning algorithms (scikit-learn, TensorFlow).

[1054] Monthly aggregation: The classified data is aggregated by month to calculate the total expenditure amount for each category.

[1055] Emotion Recognition: Analyzes user interactions and input data and recognizes user emotions with the emotion engine. Emotion recognition is performed using Microsoft Azure Cognitive Services or Google Cloud Vision API.

[1056] Customized Advice: Customize spending advice based on recognized emotions. Customized advice is generated using recommendation algorithms.

[1057] 3. User Actions

[1058] Data confirmation: Users can check the aggregated expenditure data through the app. Users can view monthly expenditure status and breakdown by category.

[1059] Get advice: Users receive personalized spending advice based on their emotions.

[1060] Specific examples

[1061] Example 1

[1062] The user goes shopping at the grocery store and takes a photo of the receipt they receive at the register. The app uses OCR to classify it as a "food expense" and, after analyzing their past purchase history and emotional data, displays advice such as, "Your food expenses have increased compared to last month, so why not take advantage of the weekend sales?"

[1063] Example prompts for generative AI models

[1064] After a user shops at a physical store, they take a photo of the receipt with the app. The app then performs OCR processing to analyze the receipt's contents. The OCR-extracted text data is then categorized and aggregated by month. The app also analyzes the user's emotions and provides personalized spending advice based on their emotions.

[1065] In this way, the system of the present invention enhances the shopping experience in brick-and-mortar stores and gives customers peace of mind in managing their spending.

[1066] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1067] Step 1:

[1068] Launching the app

[1069] A user enters a physical store and launches the app on their smartphone, which has integrated photo-taking and emotion recognition functions.

[1070] Input: None

[1071] Output: App launch

[1072] Step 2:

[1073] Taking a photo of the receipt

[1074] A user purchases an item at a store and takes a photo of the receipt they receive at the cash register using the app's camera function.

[1075] Input: Receipt

[1076] Output: Receipt image data

[1077] Step 3:

[1078] Sending image data

[1079] The device encrypts the image of the receipt it captures and sends it to a cloud server via the Internet.

[1080] Input: Receipt image data

[1081] Output: Image data sent to the server

[1082] Step 4:

[1083] Receiving image data

[1084] The server receives the image data sent from the terminal and temporarily stores it.

[1085] Input: Image data sent

[1086] Output: Image data saved in temporary storage

[1087] Step 5:

[1088] OCR processing

[1089] The server performs OCR processing on the image data received and extracts the text information written on the receipt. An OCR library such as Tesseract is used.

[1090] Input: Image data

[1091] Output: Extracted character data

[1092] Step 6:

[1093] Data Preprocessing

[1094] The server corrects misrecognition and formatting inconsistencies in the text data extracted by the OCR process, for example, correcting misrecognized letters and numbers and arranging them into a consistent format.

[1095] Input: Extracted character data

[1096] Output: Preprocessed character data

[1097] Step 7:

[1098] Categorization

[1099] The server classifies the preprocessed text data into specific categories such as food, daily necessities, medical expenses, etc. It uses machine learning algorithms such as scikit-learn and TensorFlow.

[1100] Input: Preprocessed character data

[1101] Output: Categorized data

[1102] Step 8:

[1103] Monthly tally

[1104] The server aggregates the classified data by month and performs data calculations to calculate the total expenditure amount for each category.

[1105] Input: Categorical data

[1106] Output: Monthly aggregated data

[1107] Step 9:

[1108] emotion recognition

[1109] The server analyzes the user's interactions and input data while using the app, and performs data calculations to recognize the user's emotions using an emotion engine (such as Microsoft Azure Cognitive Services or Google Cloud Vision API). It analyzes the user's facial expressions and vocabulary patterns.

[1110] Input: Interaction data and input data

[1111] Output: Emotion recognition result

[1112] Step 10:

[1113] Customizing Advice

[1114] The server generates customized spending advice based on the perceived emotions, using recommendation algorithms such as collaborative filtering.

[1115] Input: Emotion recognition results and monthly aggregated data

[1116] Output: Customized spending advice

[1117] Step 11:

[1118] Data confirmation

[1119] Users can access aggregated spending data and personalized advice through the app. Users can view monthly spending and a breakdown by category.

[1120] Input: Monthly aggregated data and customized advice

[1121] Output: Spending data and advice displayed in a user interface

[1122] In this way, the system helps users manage their spending through a series of steps and provides optimal emotional advice.

[1123] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1124] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1125] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1126] [Third embodiment]

[1127] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1128] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1129] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1130] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1131] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1132] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1133] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1134] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1135] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1136] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1137] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1138] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1139] A specific embodiment for implementing a receipt management system according to the present invention will now be described.

[1140] The system begins when a user takes a photo of a receipt using a device such as a smartphone or tablet and sends the image to a server. The server then analyzes the received image data and extracts the necessary information using optical character recognition (OCR) technology. The extracted text data is then classified by category, and monthly expenditures are calculated based on this. Finally, the aggregated data is saved in a format that can be viewed and checked by the user.

[1141] Program processing

[1142] Terminal side processing

[1143] 1. Launch the app

[1144] A user launches a receipt management app, which has a camera function and a server communication function.

[1145] 2. Take a photo of your receipt

[1146] The user takes a photo of the receipt using the app's camera function, and the image is saved in temporary storage on the device.

[1147] 3. Sending image data

[1148] The device encrypts the photo data and sends it to a server over the Internet, ensuring secure communication.

[1149] Server-side processing

[1150] 4. Receiving image data

[1151] The server receives and stores image data sent from the device, and the received data is automatically registered in the processing queue.

[1152] 5. OCR Processing

[1153] The server performs optical character recognition (OCR) on the image data to extract the text information written on the receipt, which is then saved as text data.

[1154] 6. Data Preprocessing

[1155] The server corrects misrecognition and formatting inconsistencies in character data extracted by OCR processing.

[1156] 7. Categorization

[1157] The server uses the preprocessed text data to classify expenses into categories such as food expenses, utility expenses, and medical expenses, using a machine learning algorithm.

[1158] 8. Monthly tally

[1159] The server aggregates the classified data by month and calculates the total expenditure amount and expenditure amount by category.

[1160] 9. Data Retention

[1161] The collected data is stored digitally so that users can check it later.

[1162] User behavior

[1163] 10. Data Verification

[1164] Users can check the aggregated spending data through the app, and can view monthly spending status and breakdowns by category.

[1165] 11. Receiving Advice

[1166] The user receives advice from the system. For example, if this month's food expenses have increased compared to last month, the app will suggest "reviewing your food expenses."

[1167] Specific examples

[1168] Example 1

[1169] A user goes shopping at a supermarket and receives a receipt.

[1170] The user launches the app and takes a photo of the receipt.

[1171] The device sends the captured image to the server.

[1172] The server analyzes the received image using OCR processing and extracts the text data.

[1173] The server classifies the extracted data as food expenses and reflects this in the monthly total amount.

[1174] Users can check their total food expenses through the app, which helps them manage their future spending.

[1175] Example 2

[1176] The user visits a hospital and receives a receipt.

[1177] The user uses the app to take a photo of the receipt.

[1178] The device sends the photo to the server.

[1179] The server analyzes the images and classifies them as medical expenses.

[1180] The server compiles medical expense data by month and organizes it as data for tax returns.

[1181] Users can check this month's medical expenses using the app, which can be useful when filing their tax returns.

[1182] In this way, the system of the present invention provides a means for efficiently managing receipts and visualizing expenditures.

[1183] The processing flow will be explained below.

[1184] Program processing

[1185] Step 1:

[1186] Subject: User

[1187] The user launches a receipt management app on their smartphone or tablet. The app has a photo capture function and a server communication function.

[1188] Step 2:

[1189] Subject: User

[1190] The user takes a photo of the receipt using the app's camera function, and the image is saved in the device's temporary storage.

[1191] Step 3:

[1192] Subject: Terminal

[1193] The device encrypts the captured image data, ensuring the security of the data.

[1194] Step 4:

[1195] Subject: Terminal

[1196] The device sends the encrypted image data to a server via the Internet using a secure communication protocol.

[1197] Step 5:

[1198] Subject: Server

[1199] The server receives the image data sent from the terminal and temporarily stores it. The received data is automatically added to the processing queue.

[1200] Step 6:

[1201] Subject: Server

[1202] The server performs optical character recognition (OCR) on the image data it receives, extracting the text information written on the receipt and saving it as text data.

[1203] Step 7:

[1204] Subject: Server

[1205] The server performs pre-processing to correct misrecognition and formatting inconsistencies in the character data extracted by OCR, thereby improving the accuracy of the data.

[1206] Step 8:

[1207] Subject: Server

[1208] The server classifies the preprocessed text data into specific categories (e.g., food expenses, utility bills, medical expenses) using machine learning algorithms.

[1209] Step 9:

[1210] Subject: Server

[1211] The server aggregates the categorized data by month and calculates the total expenditure for each category. The aggregated results are stored in a database and converted into a format that can be viewed by the user.

[1212] Step 10:

[1213] Subject: Server

[1214] The server organizes the results into the format required for tax returns and stores specific information such as medical expenses and donations separately.

[1215] Step 11:

[1216] Subject: User

[1217] Users can check the aggregated spending data through the app. Users can view monthly spending status and breakdowns by category.

[1218] Step 12:

[1219] Subject: User

[1220] The user receives advice from the system. For example, if food expenses have increased from last month, the user will receive a notification to "consider reviewing food expenses."

[1221] In this way, by performing specific processing for each step, the system efficiently manages receipts and makes expenditures visible.

[1222] Example 1

[1223] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1224] In recent years, receipt management has become increasingly complex, and there is a growing need to efficiently process, classify, and aggregate large volumes of receipts. Furthermore, to utilize receipt information for tax returns and expense management, it is essential to accurately and quickly extract data, correct misidentifications, and classify it into the necessary categories. Furthermore, to make it easier for users to manage their daily expenses, a system that allows easy data storage and viewing and provides appropriate advice is required.

[1225] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1226] In this invention, the server includes optical character recognition means for extracting character data from received images, means using a machine learning algorithm for classifying the extracted character data into specific categories, and preprocessing means for correcting misrecognition and formatting inconsistencies in the extracted character data. This allows for accurate extraction of text data from receipt images, classification into appropriate categories, and monthly aggregation. The system also includes means for organizing and saving information necessary for tax returns, thereby streamlining expense management and tax processing. Furthermore, a function for providing advice to users facilitates expense management.

[1227] The "means for taking a picture of a receipt" is a function that allows a user to take a picture of a receipt using a terminal.

[1228] The "means for transmitting the photographed image of the receipt to the server" is a function that enables the terminal to encrypt the photographed image data and transfer it to the server via the Internet.

[1229] "Optical character recognition means for extracting character data from received images" is a technology in which the server analyzes image data and extracts character information written on the receipt as text data.

[1230] "Means using a machine learning algorithm to classify extracted character data into specific categories" refers to an algorithm that the server uses machine learning technology to classify extracted character data into categories such as food expenses, utility expenses, and medical expenses.

[1231] The "means for aggregating classified data by month" is a function that allows the server to calculate the total monthly expenditure and the expenditure by category based on the data classified by category.

[1232] "Means for storing aggregated data and providing it to users" refers to the function that enables the server to safely store the aggregated data and allow users to view and check it later.

[1233] "Preprocessing means for correcting misrecognition or formatting inconsistencies in extracted character data" refers to a process in which the server corrects characters misrecognized during OCR processing or formatting inconsistencies to generate accurate text data.

[1234] The "means for providing advice to the user" is a function by which the server provides financial advice and suggestions for spending management based on the user's spending data.

[1235] A specific embodiment for implementing a receipt management system according to the present invention will be described. The purpose of this system is to efficiently manage receipts that users receive on a daily basis and to tally and classify expenditure data.

[1236] The system starts when a user takes a picture of a receipt using a device such as a smartphone or tablet. A receipt management app is installed on this device, which integrates a camera function and a server communication function. When a user launches the app and takes a picture of a receipt using the camera function, the image is saved in temporary storage on the device.

[1237] The device then encrypts the captured photo data and sends it over the Internet to a server using AES (Advanced Encryption Standard) encryption and HTTPS (Hyper Text Transfer Protocol Secure) to ensure secure communications.

[1238] The server receives the image data sent from the device and securely stores it in storage. The received data is automatically queued for processing. The server then analyzes the received image data using optical character recognition (OCR) technology to extract the text information listed on the receipt. Specifically, Google Cloud Vision API or other OCR software is used. The text data extracted by the OCR process may contain recognition errors or formatting inconsistencies, so the server performs preprocessing to correct these. This preprocessing includes spell checking and text formatting.

[1239] After preprocessing, the text data is then classified into specific categories using machine learning algorithms, such as libraries like scikit-learn and TensorFlow, including categories like food, utility, and medical expenses.

[1240] The classified data is aggregated monthly by the server, efficiently organized using an SQL database, and then securely stored in digital infrastructure such as Amazon S3 or Google Cloud Storage, where users can review it later.

[1241] Users can check the collected expenditure data through the app. When users press the "Check Data" button, monthly expenditure status and breakdown by category will be displayed. The system also has a function that allows users to receive financial advice. For example, if this month's food expenses have increased compared to last month, the system will suggest that users "review their food expenses."

[1242] Specific examples

[1243] Example 1

[1244] A user goes shopping at a supermarket and receives a receipt.

[1245] The user launches the app and takes a photo of the receipt.

[1246] The images captured by the device are encrypted using AES and sent to the server.

[1247] The server processes the images received using the Google Cloud Vision API and extracts text data.

[1248] The server classifies the extracted data as food expenses and reflects this in the monthly total amount.

[1249] The user checks the total food expenses through the app and receives the advice, "This month's food expenses have increased by 20% compared to last month."

[1250] Example 2

[1251] The user visits a hospital and receives a receipt.

[1252] The user uses the app to take a photo of the receipt.

[1253] The device encrypts the photo and sends it to the server.

[1254] The server analyzes the image using OCR processing and classifies it as medical expenses.

[1255] The server compiles medical expense data by month and organizes it as data for tax returns.

[1256] The user checks this month's medical expenses in the app and receives advice that "medical expense deductions may be applicable when filing your tax return."

[1257] Example prompts for generative AI models

[1258] "Describe in natural language the process of a receipt management system, from a user taking a photo of a receipt to a server parsing it to generate spending data. Explain each step of the process in detail and provide two concrete examples."

[1259] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1260] Step 1: Launch the app

[1261] A user launches a receipt management app on a smartphone or tablet. The app integrates a camera function and a server communication function. The input is the user's operation, and the output is the running status of the app.

[1262] process:

[1263] The user taps the app icon to launch it.

[1264] The app will launch and display the home screen.

[1265] Step 2: Take a photo of your receipt

[1266] The user takes a photo of the receipt using the camera function in the app. When the user presses the capture button, the camera takes a photo and the image is saved in temporary storage on the device. The input is the execution of the camera function, and the output is the captured image.

[1267] process:

[1268] The user presses the "Take Photo" button.

[1269] The device activates the camera, and the user frames the receipt and then presses the shutter button.

[1270] The image is saved to temporary storage.

[1271] Step 3: Sending image data

[1272] The device encrypts the captured photo data and sends it to a server over the Internet. This process uses AES encryption and the HTTPS protocol. The input is the image data, and the output is the encrypted data and its transmission.

[1273] process:

[1274] The device reads the image data from temporary storage.

[1275] Encrypt image data using AES.

[1276] The encrypted data is sent to the server using HTTPS.

[1277] Step 4: Receiving image data

[1278] The server receives image data sent from the device and stores it securely in storage. The received data is automatically registered in the processing queue. The input is encrypted image data, and the output is decrypted image data and its storage.

[1279] process:

[1280] The server receives the encrypted data using the HTTPS protocol.

[1281] The received data is decrypted using AES.

[1282] The decrypted data is stored in the server's storage and queued for processing.

[1283] Step 5: OCR processing

[1284] The server analyzes the received image data using optical character recognition (OCR) technology and extracts text information. Specifically, OCR software is used. The input is the decoded image data, and the output is the extracted text data.

[1285] process:

[1286] The server calls the Google Cloud Vision API or other OCR software to analyze the image data.

[1287] The text information written on the receipt is extracted using OCR processing.

[1288] The extracted text data is temporarily saved.

[1289] Step 6: Preprocessing the data

[1290] The server corrects misrecognition and formatting inconsistencies in the character data extracted by the OCR process. This process checks the spelling and formatting of the text data. The input is the extracted text data, and the output is the corrected text data.

[1291] process:

[1292] The server performs a spell check on the extracted text data.

[1293] Detect and correct formatting inconsistencies.

[1294] The corrected text data is saved.

[1295] Step 7: Categorize

[1296] The server classifies the preprocessed text data into categories using machine learning algorithms. The input is the corrected text data, and the output is the data classified into categories.

[1297] process:

[1298] The server invokes the machine learning model using libraries such as scikit-learn and TensorFlow.

[1299] The preprocessed data is fed into the model and categorical labels are assigned.

[1300] Store classified data.

[1301] Step 8: Monthly Tally

[1302] The server aggregates the classified data by month and calculates the total expenditure amount and expenditure amount by category. The input is data classified by category, and the output is the aggregated data.

[1303] process:

[1304] The server aggregates the categorized data by month using a database query.

[1305] Calculate total expenses and expenses by category for each month.

[1306] Save the summary results.

[1307] Step 9: Save your data

[1308] The server stores the aggregated data for users to review later. The input is the aggregated data, and the output is the stored data.

[1309] process:

[1310] The server stores the data on a digital platform such as Amazon S3 or Google Cloud Storage.

[1311] Verify that the saved data is stored correctly.

[1312] Step 10: Data verification

[1313] The user checks the aggregated spending data through the app. The input is the user request, and the output is the displayed spending data.

[1314] process:

[1315] The user presses the "Check Data" button in the app.

[1316] The server sends the stored aggregated data to the app in response to a user request.

[1317] The device displays the received data, allowing the user to check monthly spending and breakdowns by category.

[1318] Step 11: Getting advice

[1319] The user receives advice from the system: the input is expenditure data and the output is an advice message.

[1320] process:

[1321] The server analyzes spending data to detect unusual patterns and areas for improvement.

[1322] An advice message is generated and notified to the user.

[1323] Users can review advice messages to help them manage their future spending.

[1324] (Application example 1)

[1325] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1326] Traditional receipt management systems require the manual management of paper receipts, which is time-consuming and labor-intensive. They also lack the ability to efficiently import electronic receipts, making overall spending management cumbersome. They also make it difficult for users to understand their spending trends and identify areas for improvement. Furthermore, extracted character data can be misrecognized or incompletely classified, making accurate spending management difficult.

[1327] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1328] In this invention, the server includes a means for transmitting an image of a receipt to the server, an optical character recognition means for extracting character data from the received image, a means for classifying the extracted character data into specific categories, a means for automatically capturing electronic receipts and transmitting them to the system, and a means for providing the user with spending advice based on the compiled data. This allows users to efficiently manage not only paper receipts but also electronic receipts, making it easy to understand spending trends and areas for improvement. Furthermore, accurate classification of the extracted character data makes user spending management more accurate and efficient.

[1329] "Means for taking an image of a receipt" refers to a mobile device equipped with a camera function for capturing a digital image of a paper receipt.

[1330] "Means for transmitting the photographed image of the receipt to the server" refers to a communication function for transferring the digital image data of the acquired receipt to the server via a network.

[1331] "Optical character recognition means" means optical character recognition (OCR) technology for identifying characters from a digital image and extracting them as text data.

[1332] "Means for classifying extracted character data into specific categories" refers to algorithms or machine learning models that automatically organize and classify text data obtained through OCR into specific categories, such as food expenses and medical expenses.

[1333] "Means for aggregating classified data by month" refers to a program that aggregates data classified into specific categories on a monthly basis and calculates total expenditures and expenditures by category.

[1334] "Means for storing aggregated data and providing it to users" refers to the interface that stores monthly aggregated data in the server and allows users to check and view that data.

[1335] "Means for automatically receiving and transmitting electronic receipts to the system" refers to a function for automatically receiving electronic receipts generated from electronic payment services and managing them on the system.

[1336] "Means for providing spending advice to users based on aggregated data" refers to functionality that analyzes aggregated data and makes suggestions for improving or managing a user's spending habits.

[1337] This invention begins when a user takes a photo of a receipt using a device such as a smartphone and sends the data to a server. The server then analyzes the received image data using optical character recognition (OCR) technology, classifies the extracted text data by category, and aggregates it by month. This system allows users to efficiently manage their expenses. It also has a function that links with electronic payment services to automatically import electronic receipts and send them to the system.

[1338] Hardware and Software

[1339] Hardware:

[1340] Smartphone or tablet camera: Used to take a picture of the receipt.

[1341] Server: Used to receive, analyze, and store image data.

[1342] software:

[1343] OpenCV: Used for image preprocessing (grayscale conversion and binarization).

[1344] pytesseract: A library for OCR processing, used to extract text data from images.

[1345] scikit-learn: Used to classify character data extracted by machine learning algorithms into categories.

[1346] Program processing explanation

[1347] The server first receives the image data sent by the user and preprocesses the image using OpenCV. Next, it uses pytesseract to extract text data from the preprocessed image. It then uses the scikit-learn machine learning algorithm to classify the extracted text data into specific categories. The classified data is aggregated by month, and the total expenditure amount and expenditure amount by category are calculated. The results are stored on the server and made available for users to view and check through the application.

[1348] The company has also introduced a system for automatically capturing digital receipts, allowing users to manage both paper and digital receipts in one place, and it also has a function that provides users with spending advice based on the aggregated data.

[1349] Examples and prompts

[1350] Example 1

[1351] Let's say a user goes shopping at a supermarket and receives a receipt. When the user launches the smartphone app and takes a photo of the receipt, the device sends the image to the server. The server analyzes the received image using OCR processing and extracts text data. The system then classifies the extracted data as food expenses and reflects it in the monthly total. The user can check the total food expenses through the app, which can be used to manage future expenses.

[1352] Example 2

[1353] When a user receives a medical receipt from a hospital, they use the app to take a photo of the receipt, and the device sends the photo to the server. The server analyzes the image and classifies it as medical expenses. This data is aggregated monthly and organized as tax return data. Users can check their medical expenses for the month in the app, which can be useful when filing their tax return.

[1354] Prompt Sentence Examples

[1355] "A user's food spending this month has increased by 20% compared to last month. Please provide advice on how to address this increase."

[1356] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1357] Step 1:

[1358] The user launches the application and takes a picture of the receipt. When the user takes a picture of a paper receipt using a smartphone or tablet, the image is temporarily saved on the device. The input is the image of the receipt, and the output is the temporarily saved digital image data.

[1359] Step 2:

[1360] The device sends the captured image data to the server. The device transfers the encrypted image data to the server via the Internet. The input is the temporarily stored digital image data, and the output is the image data sent to the server.

[1361] Step 3:

[1362] The server stores the received image data and performs preprocessing for OCR. The server uses OpenCV to convert the image to grayscale and perform binarization. The input is the image data sent to the server, and the output is the preprocessed image data.

[1363] Step 4:

[1364] The server performs OCR processing on the preprocessed image and extracts character data. The server extracts the text data using pytesseract and saves it as character data in UTF-8 format. The input is the preprocessed image data, and the output is the extracted character data.

[1365] Step 5:

[1366] The server preprocesses the extracted character data to check for misrecognitions and inconsistencies. The server uses regular expressions to clean the extracted data and correct misrecognitions. The input is the extracted character data, and the output is the preprocessed, clean character data.

[1367] Step 6:

[1368] The server classifies the preprocessed text data into specific categories. The server uses scikit-learn's machine learning algorithm to automatically classify the data into categories such as food expenses and medical expenses. The input is the preprocessed, clean text data, and the output is the classified category data.

[1369] Step 7:

[1370] The server aggregates the classified data by month and calculates the total expenditure amount and expenditure amount by category. The server uses an SQL database to store and aggregate the data and organize it by month. The input is the classified category data, and the output is the expenditure data aggregated by month.

[1371] Step 8:

[1372] The server stores the aggregated data and provides it to the user. The server stores the aggregated results in a database and allows the user to view the data through the app. The input is monthly aggregated expenditure data, and the output is the stored aggregated data and the data displayed in the user interface.

[1373] Step 9:

[1374] The server provides spending advice to the user based on the aggregated data. The server uses a generative AI model to analyze the user's spending data and generate advice based on prompts. The input is monthly aggregated spending data and prompts, and the output is the generated advice.

[1375] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1376] A specific embodiment of the receipt management system combined with an emotion engine according to the present invention will now be described.

[1377] The system begins when a user takes a photo of a receipt using a device such as a smartphone or tablet and sends the image to a server. The server analyzes the received image data and extracts the necessary information using optical character recognition (OCR). The extracted text data is classified by category, and monthly expenditures are calculated based on this. The system then uses an emotion engine to recognize the user's emotions and customize spending advice based on this.

[1378] Program processing

[1379] Terminal side processing

[1380] 1. Launch the app

[1381] A user launches a receipt management app that integrates photo capture, emotion recognition, and server communication functions.

[1382] 2. Take a photo of your receipt

[1383] The user takes a photo of the receipt using the app's camera function, and the image is saved in the device's temporary storage.

[1384] 3. Sending image data

[1385] The device encrypts the photo data and sends it to a server over the Internet, ensuring secure communication.

[1386] Server-side processing

[1387] 4. Receiving image data

[1388] The server receives the image data sent from the terminal and temporarily stores it. The received data is automatically added to the processing queue.

[1389] 5. OCR Processing

[1390] The server performs optical character recognition (OCR) on the image data, extracting the text information written on the receipt and saving it as text data.

[1391] 6. Data Preprocessing

[1392] The server performs pre-processing to correct misrecognition and formatting inconsistencies in the character data extracted by OCR, thereby improving the accuracy of the data.

[1393] 7. Categorization

[1394] The server classifies the preprocessed text data into specific categories (e.g., food expenses, utility bills, medical expenses) using machine learning algorithms.

[1395] 8. Monthly tally

[1396] The server aggregates the categorized data by month and calculates the total expenditure for each category. The aggregated results are stored in a database and converted into a format that can be viewed by the user.

[1397] 9. Emotion recognition

[1398] The server uses an emotion engine to recognize the user's emotions from the user's input data and interaction logs, for example, by analyzing facial expressions and usage patterns displayed while the user is using the app.

[1399] 10. Customizing Advice

[1400] The server then customizes spending advice based on the perceived emotion, for example, gently suggesting savings advice if the user is feeling stressed.

[1401] 11. Data Retention

[1402] The aggregated data and customized advice will be stored digitally for users to review at a later date.

[1403] User behavior

[1404] 12. Data Verification

[1405] Users can check the aggregated spending data through the app. Users can view monthly spending status and breakdowns by category.

[1406] 13. Receiving Advice

[1407] The user receives customized advice from the system, for example, if their food expenses have increased since last month, the system will display advice tailored to their current emotional state.

[1408] Specific examples

[1409] Example 1

[1410] A user goes shopping at a supermarket and receives a receipt.

[1411] The user launches the app and takes a photo of the receipt.

[1412] The device sends the captured image to the server.

[1413] The server analyzes the received image using OCR processing and extracts the text data.

[1414] The server classifies the extracted data as food expenses and reflects this in the monthly total amount.

[1415] The server recognizes the user's emotions and makes gentle suggestions to save money if stress levels are high.

[1416] Users can check the total food cost and advice through the app to help manage their future spending.

[1417] Example 2

[1418] The user visits a hospital and receives a receipt.

[1419] The user uses the app to take a photo of the receipt.

[1420] The device sends the photo to the server.

[1421] The server analyzes the images and classifies them as medical expenses.

[1422] The server compiles medical expense data by month and organizes it as data for tax returns.

[1423] The server analyzes the user's emotions and, if anxiety is high, provides reassuring advice on reviewing spending, for example.

[1424] Users can check this month's medical expenses using the app, which can be useful when filing their tax returns.

[1425] In this way, the system of the present invention efficiently manages receipts and visualizes expenditures, and by combining it with an emotion engine, it provides expenditure advice tailored to the user's emotional state.

[1426] The processing flow will be explained below.

[1427] Program processing

[1428] Step 1:

[1429] Subject: User

[1430] A user launches a receipt management app on a smartphone or tablet, which is equipped with a camera, emotion recognition, and server communication functions.

[1431] Step 2:

[1432] Subject: User

[1433] The user takes a photo of the receipt using the app's camera function, and the image is saved in the device's temporary storage.

[1434] Step 3:

[1435] Subject: Terminal

[1436] The device encrypts the captured image data, ensuring the security of the data.

[1437] Step 4:

[1438] Subject: Terminal

[1439] The device sends the encrypted image data to a server via the Internet using a secure communication protocol.

[1440] Step 5:

[1441] Subject: Server

[1442] The server receives the image data sent from the terminal and temporarily stores it. The received data is automatically added to the processing queue.

[1443] Step 6:

[1444] Subject: Server

[1445] The server performs optical character recognition (OCR) on the image data it receives, extracting the text information written on the receipt and saving it as text data.

[1446] Step 7:

[1447] Subject: Server

[1448] The server performs pre-processing to correct misrecognition and formatting inconsistencies in the character data extracted by OCR, thereby improving the accuracy of the data.

[1449] Step 8:

[1450] Subject: Server

[1451] The server classifies the preprocessed text data into specific categories (e.g., food expenses, utility bills, medical expenses) using machine learning algorithms.

[1452] Step 9:

[1453] Subject: Server

[1454] The server aggregates the categorized data by month and calculates the total expenditure for each category. The aggregated results are stored in a database and converted into a format that can be viewed by the user.

[1455] Step 10:

[1456] Subject: Server

[1457] The server uses an emotion engine to recognize the user's emotions from the user's input data and interaction logs, for example, by analyzing facial expressions and usage patterns displayed while the user is using the app.

[1458] Step 11:

[1459] Subject: Server

[1460] The server then customizes spending advice based on the perceived emotion, for example, gently suggesting savings advice if the user is feeling stressed.

[1461] Step 12:

[1462] Subject: Server

[1463] The server then stores the results and customized advice digitally, allowing users to review them later.

[1464] Step 13:

[1465] Subject: User

[1466] Users can check the aggregated spending data through the app. Users can view monthly spending status and breakdowns by category.

[1467] Step 14:

[1468] Subject: User

[1469] The user receives customized advice from the system, for example, if their food expenses have increased since last month, the system will display advice tailored to their current emotional state.

[1470] Example 2

[1471] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1472] In modern society, managing and organizing the large number of receipts generated in daily life and business situations is a significant burden for users. Taking images of receipts, extracting information, and accurately categorizing and aggregating them is particularly time-consuming and labor-intensive. Furthermore, since appropriate spending advice is not provided based on the user's emotional state, it can easily lead to stressful situations. Conventional technologies have not been able to adequately resolve these issues.

[1473] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1474] In this invention, the server includes means for capturing an image of a receipt, means for transmitting the captured image of the receipt to the server, optical character recognition means for extracting character data from the received image, means for classifying the extracted character data into specific categories, means for aggregating the classified data by month, means for recognizing a user's emotion, means for customizing advice on spending based on the recognized emotion, and means for saving the aggregated data and customized advice and providing it to the user. This not only improves the efficiency of receipt management work but also enables spending advice tailored to the user's emotional state.

[1475] A "receipt" is a document issued to prove the content and amount of a transaction when purchasing a product or using a service.

[1476] An "image" is visual information recorded in digital form, and is data that includes text, graphics, etc.

[1477] A "server" is a computer system that provides data and services to other devices (terminals) within a network.

[1478] A "terminal" is a communication device or computer that can only function when connected to a network.

[1479] Optical character recognition (OCR) is a technology that reads character information in an image and converts it into digital text.

[1480] A "category" is a division for classifying data according to its type or characteristics.

[1481] "Aggregation" is the process of aggregating multiple data sets and calculating statistical information such as totals and averages.

[1482] "Emotion recognition" is a technology that analyzes and determines a user's emotional state from data.

[1483] "Advice" is advice or suggestions offered to encourage optimal action or decision in a particular situation.

[1484] "Storage" is the act of writing down data on a recording medium for later use.

[1485] "User" refers to an individual or corporation that uses this system to manage receipts and check data.

[1486] This section describes a specific example of combining an emotion engine with a receipt management system according to the present invention. This system begins when a user takes an image of a receipt using a device such as a smartphone or tablet and sends the image to a server. The specific hardware used is a smartphone or tablet, and the software used is a dedicated receipt management app.

[1487] The server-side processing includes receiving image data, OCR processing, data preprocessing, categorization, monthly aggregation, emotion recognition, and generating and providing customized advice. These processes are performed using the following technologies:

[1488] Server-side processing

[1489] 1. Receiving image data

[1490] The server receives the image data sent from the device and temporarily stores it. The HTTPS protocol is used for communication, and the AES encryption algorithm ensures data security.

[1491] 2. OCR processing

[1492] The server uses optical character recognition technology, such as Google Cloud Vision API, to extract text from the image data. This process converts the text data on the receipt into text format.

[1493] 3. Data Preprocessing

[1494] The character data extracted by OCR may contain misrecognition errors or formatting inconsistencies, and the server uses a Python-based data cleaning tool to correct these errors.

[1495] 4. Categorization

[1496] A machine learning algorithm (e.g., the BERT model) is used to classify the preprocessed text data into specific categories (e.g., food expenses, utility expenses, medical expenses), thereby achieving accurate data classification.

[1497] 5. Monthly tally

[1498] The categorized data is aggregated by month, and the total expenditure for each category is calculated. The aggregated results are stored in a database (e.g., MySQL) and can be viewed by users in HTML or JSON format.

[1499] 6. Emotion recognition

[1500] We use Microsoft Azure's Emotion API to recognize the user's emotional state from the user's input data and interaction logs, which allows us to accurately determine the user's emotional state (e.g., stress, relief).

[1501] 7. Customizing Advice

[1502] Based on the recognized emotions, spending advice is customized using a generative AI model (e.g., OpenAI's GPT-4). For example, it generates "food expenses are increasing, but reasonable savings suggestions are generated due to high stress."

[1503] 8. Data storage and provision

[1504] The aggregated data and customized advice is stored on Amazon S3 or Google Cloud Storage, which users can then review later through the app.

[1505] User behavior

[1506] Users can view aggregated spending data through the app. They can view monthly spending status and category breakdowns in graphs and tables for specific spending management. They can also receive customized spending advice, allowing them to review their spending appropriately based on their emotional state.

[1507] Specific examples

[1508] As a concrete example, let's consider the process of a user shopping at a supermarket, receiving a receipt, and then taking a photo of the receipt using the app. The device sends this photo data to a server, which then analyzes the image using OCR to extract text data. The extracted data is classified as "food expenses" and reflected in the monthly total. If the user's stress level is high using emotion recognition, gentle suggestions for saving money are made, which the user can confirm through the app and use to help manage future expenses.

[1509] Next, when the user visits a hospital for a medical examination and receives a receipt, the same procedure is followed. The app takes a photo of the receipt, and the server classifies it as a medical expense. The medical expense data is aggregated by month and organized for tax return filing. Furthermore, if the user is highly anxious based on sentiment analysis, the app provides advice on reviewing expenses to reassure them.

[1510] Example prompt using a generative AI model: "My food expenses have been increasing recently, and I'm under a lot of stress. Can you tell me some reasonable ways to save money?"

[1511] The above is a specific embodiment for implementing the present invention. This system efficiently manages receipts and visualizes expenditures, and by combining it with an emotion engine, it provides expenditure advice tailored to the user's emotional state.

[1512] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1513] Step 1:

[1514] Launching the app

[1515] The user launches the receipt management app on their smartphone or tablet. The app prompts the user for authentication information (e.g., ID and password) the first time it is launched. If authentication is successful, the user is redirected to the main screen.

[1516] Input: Launching an app through user interaction

[1517] Output: The main screen of the app is displayed.

[1518] Step 2:

[1519] Taking a photo of the receipt

[1520] The user takes a photo of the receipt using the app's camera, and the image is saved in temporary device storage in JPEG format, ready to be sent to the server later.

[1521] Input: User-taken image of receipt

[1522] Output: JPEG image data is saved to temporary storage

[1523] Step 3:

[1524] Sending image data

[1525] The device encrypts the captured image data using the 128-bit AES encryption algorithm, then sends the encrypted image data to the server using HTTPS, with the TLS protocol ensuring communication security.

[1526] Input: JPEG format image data

[1527] Output: The encrypted image data is sent to the server.

[1528] Step 4:

[1529] Receiving image data

[1530] The server receives the image data sent from the device and temporarily stores it. The received data is automatically added to the processing queue and awaits analysis.

[1531] Input: Encrypted image data

[1532] Output: Temporarily saved image data

[1533] Step 5:

[1534] OCR processing

[1535] The server uses the Google Cloud Vision API to perform OCR processing on the image data, extracting the text information from the receipt and saving it as text data.

[1536] Input: Temporarily saved image data

[1537] Output: Extracted text data

[1538] Step 6:

[1539] Data Preprocessing

[1540] The server performs pre-processing to correct misrecognition and formatting inconsistencies in the character data extracted by OCR. For example, if the number "1" is misrecognized as a "lowercase l" or an "uppercase I," it will be converted to the correct format.

[1541] Input: Extracted text data

[1542] Output: Preprocessed text data

[1543] Step 7:

[1544] Categorization

[1545] The server classifies the preprocessed text data into specific categories (e.g., food expenses, utility expenses, medical expenses) using a machine learning algorithm (e.g., the BERT model).

[1546] Input: Preprocessed text data

[1547] Output: Categorized data

[1548] Step 8:

[1549] Monthly tally

[1550] The server aggregates the categorized data by month and calculates the total expenditure for each category. The aggregated results are stored in a database (e.g., MySQL) and can be viewed by users in HTML or JSON format.

[1551] Input: Categorical data

[1552] Output: Monthly aggregated data

[1553] Step 9:

[1554] emotion recognition

[1555] The server uses Microsoft Azure's Emotion API to recognize the user's emotional state from the user's input data and interaction logs, thereby accurately determining the user's emotional state (e.g., stress, relief).

[1556] Input: User input data and interaction logs

[1557] Output: Recognized emotion data

[1558] Step 10:

[1559] Customizing Advice

[1560] The server then uses a generative AI model (e.g., OpenAI's GPT-4) to customize spending advice based on the recognized emotions. For example, it generates a reasonable saving suggestion, saying, "Food expenses are increasing, but stress is high."

[1561] Input: Recognized emotion data

[1562] Output: Customized advice

[1563] Step 11:

[1564] Data storage and provision

[1565] The server stores the aggregated data and customized advice on Amazon S3 or Google Cloud Storage, which users can then view via the app.

[1566] Input: Monthly aggregated data and customized advice

[1567] Output: Stored data and advice

[1568] Step 12:

[1569] Data confirmation

[1570] The user accesses the "Data Check" section of the app, where they can view their monthly spending and a breakdown by category in graphs and tables.

[1571] Input: Data confirmation request by user operation

[1572] Output: Screen showing expenditure status and breakdown

[1573] Step 13:

[1574] Receiving advice

[1575] The user accesses the "Advice" section and sees customized spending advice from the system. For example, because food expenses have increased since last month, soft suggestions are displayed that are tailored to the user's emotional state.

[1576] Input: User initiated advice request

[1577] Output: Screen showing customized advice

[1578] (Application example 2)

[1579] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1580] In today's brick-and-mortar shopping environment, customers lack the means to efficiently manage receipts and understand their spending in real time. Furthermore, when customers experience emotional stress related to their spending habits, they often lack the ability to receive appropriate advice and support. There is a need for a system that can solve these problems and provide more personalized spending management and psychological support to customers.

[1581] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1582] In this invention, the server includes means for capturing an image of a receipt, means for transmitting the captured image of the receipt to the server, optical character recognition means for extracting character data from the received image, means for classifying the extracted character data into specific categories, means for aggregating the classified data by month, means for storing and providing the aggregated data to a user, and means for recognizing the user's emotions and customizing spending advice based on the emotion data, thereby enabling customers to manage their spending in real time while shopping at a physical store and receive optimal spending advice based on their emotions.

[1583] The "means for taking a picture of a receipt" is a function that allows a user to take a picture of a receipt for a product purchased at a physical store using the camera on a mobile device such as a smartphone.

[1584] The "means for transmitting the photographed image of the receipt to the server" is a communication function that encrypts the photographed image of the receipt and transmits it securely to the server.

[1585] "Optical character recognition means for extracting character data from received images" refers to a function that uses optical character recognition (OCR) technology to analyze character information from receipt images received by the server and extract it as text data.

[1586] The "means for classifying extracted character data into specific categories" is a function for classifying character data extracted by OCR into specific categories such as food, daily necessities, and medical expenses.

[1587] The "means for aggregating classified data by month" is a function for aggregating expenditure data classified by category by month.

[1588] The "means for storing aggregated data and providing it to the user" is a function for storing the aggregated results in a database and displaying them so that the user can check them through the app.

[1589] "Means for recognizing user emotions and customizing spending advice based on this emotional data" refers to a function that analyzes user interactions and input data to recognize emotions and provides individually optimized spending advice based on the results of this recognition.

[1590] This invention is a system that allows customers in physical stores to use their smartphones to manage receipts and visualize spending in real time, and provides personalized advice based on their emotions. Specific embodiments of the invention are described below.

[1591] System Overview

[1592] It consists of three elements: a server, a terminal (such as a smartphone), and a user. Details of each element are as follows:

[1593] Required Hardware and Software

[1594] Hardware: Smartphone (compatible with Android and iOS), store Wi-Fi network, cloud server

[1595] Software: smartphone applications, OCR libraries (e.g., Tesseract), emotion recognition algorithms (e.g., Microsoft Azure Cognitive Services or Google Cloud Vision API)

[1596] Processing flow and data processing

[1597] 1. Terminal processing

[1598] App launch: A user enters a store and launches the app, which has integrated photo-taking and emotion recognition functions.

[1599] Taking a photo of a receipt: The user uses the camera function within the app to take a photo of the receipt. The captured image is saved in the device's temporary storage.

[1600] Transmission of image data: The device encrypts the image data and transmits it to a server via the Internet.

[1601] 2. Server-side processing

[1602] Receiving image data: The server receives the image data sent from the terminal and temporarily stores it.

[1603] OCR processing: The server performs OCR processing on the image data received and extracts the text information written on the receipt.

[1604] Data preprocessing: Preprocessing is performed to correct misrecognition and formatting inconsistencies in the character data extracted by OCR processing.

[1605] Category Classification: Classify the preprocessed text data into specific categories such as food, daily necessities, medical expenses, etc., using machine learning algorithms (scikit-learn, TensorFlow).

[1606] Monthly aggregation: The classified data is aggregated by month to calculate the total expenditure amount for each category.

[1607] Emotion Recognition: Analyzes user interactions and input data and recognizes user emotions with the emotion engine. Emotion recognition is performed using Microsoft Azure Cognitive Services or Google Cloud Vision API.

[1608] Customized Advice: Customize spending advice based on recognized emotions. Customized advice is generated using recommendation algorithms.

[1609] 3. User Actions

[1610] Data confirmation: Users can check the aggregated expenditure data through the app. Users can view monthly expenditure status and breakdown by category.

[1611] Get advice: Users receive personalized spending advice based on their emotions.

[1612] Specific examples

[1613] Example 1

[1614] The user goes shopping at the grocery store and takes a photo of the receipt they receive at the register. The app uses OCR to classify it as a "food expense" and, after analyzing their past purchase history and emotional data, displays advice such as, "Your food expenses have increased compared to last month, so why not take advantage of the weekend sales?"

[1615] Example prompts for generative AI models

[1616] After a user shops at a physical store, they take a photo of the receipt with the app. The app then performs OCR processing to analyze the receipt's contents. The OCR-extracted text data is then categorized and aggregated by month. The app also analyzes the user's emotions and provides personalized spending advice based on their emotions.

[1617] In this way, the system of the present invention enhances the shopping experience in brick-and-mortar stores and gives customers peace of mind in managing their spending.

[1618] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1619] Step 1:

[1620] Launching the app

[1621] A user enters a physical store and launches the app on their smartphone, which has integrated photo-taking and emotion recognition functions.

[1622] Input: None

[1623] Output: App launch

[1624] Step 2:

[1625] Taking a photo of the receipt

[1626] A user purchases an item at a store and takes a photo of the receipt they receive at the cash register using the app's camera function.

[1627] Input: Receipt

[1628] Output: Receipt image data

[1629] Step 3:

[1630] Sending image data

[1631] The device encrypts the image of the receipt it captures and sends it to a cloud server via the Internet.

[1632] Input: Receipt image data

[1633] Output: Image data sent to the server

[1634] Step 4:

[1635] Receiving image data

[1636] The server receives the image data sent from the terminal and temporarily stores it.

[1637] Input: Image data sent

[1638] Output: Image data saved in temporary storage

[1639] Step 5:

[1640] OCR processing

[1641] The server performs OCR processing on the image data received and extracts the text information written on the receipt. An OCR library such as Tesseract is used.

[1642] Input: Image data

[1643] Output: Extracted character data

[1644] Step 6:

[1645] Data Preprocessing

[1646] The server corrects misrecognition and formatting inconsistencies in the text data extracted by the OCR process, for example, correcting misrecognized letters and numbers and arranging them into a consistent format.

[1647] Input: Extracted character data

[1648] Output: Preprocessed character data

[1649] Step 7:

[1650] Categorization

[1651] The server classifies the preprocessed text data into specific categories such as food, daily necessities, medical expenses, etc. It uses machine learning algorithms such as scikit-learn and TensorFlow.

[1652] Input: Preprocessed character data

[1653] Output: Categorized data

[1654] Step 8:

[1655] Monthly tally

[1656] The server aggregates the classified data by month and performs data calculations to calculate the total expenditure amount for each category.

[1657] Input: Categorical data

[1658] Output: Monthly aggregated data

[1659] Step 9:

[1660] emotion recognition

[1661] The server analyzes the user's interactions and input data while using the app, and performs data calculations to recognize the user's emotions using an emotion engine (such as Microsoft Azure Cognitive Services or Google Cloud Vision API). It analyzes the user's facial expressions and vocabulary patterns.

[1662] Input: Interaction data and input data

[1663] Output: Emotion recognition result

[1664] Step 10:

[1665] Customizing Advice

[1666] The server generates customized spending advice based on the perceived emotions, using recommendation algorithms such as collaborative filtering.

[1667] Input: Emotion recognition results and monthly aggregated data

[1668] Output: Customized spending advice

[1669] Step 11:

[1670] Data confirmation

[1671] Users can access aggregated spending data and personalized advice through the app. Users can view monthly spending and a breakdown by category.

[1672] Input: Monthly aggregated data and customized advice

[1673] Output: Spending data and advice displayed in a user interface

[1674] In this way, the system helps users manage their spending through a series of steps and provides optimal emotional advice.

[1675] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1676] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1677] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1678] [Fourth embodiment]

[1679] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1680] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1681] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1682] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1683] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1684] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1685] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1686] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1687] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1688] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1689] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1690] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1691] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1692] A specific embodiment for implementing a receipt management system according to the present invention will now be described.

[1693] The system begins when a user takes a photo of a receipt using a device such as a smartphone or tablet and sends the image to a server. The server then analyzes the received image data and extracts the necessary information using optical character recognition (OCR) technology. The extracted text data is then classified by category, and monthly expenditures are calculated based on this. Finally, the aggregated data is saved in a format that can be viewed and checked by the user.

[1694] Program processing

[1695] Terminal side processing

[1696] 1. Launch the app

[1697] A user launches a receipt management app, which has a camera function and a server communication function.

[1698] 2. Take a photo of your receipt

[1699] The user takes a photo of the receipt using the app's camera function, and the image is saved in temporary storage on the device.

[1700] 3. Sending image data

[1701] The device encrypts the photo data and sends it to a server over the Internet, ensuring secure communication.

[1702] Server-side processing

[1703] 4. Receiving image data

[1704] The server receives and stores image data sent from the device, and the received data is automatically registered in the processing queue.

[1705] 5. OCR Processing

[1706] The server performs optical character recognition (OCR) on the image data to extract the text information written on the receipt, which is then saved as text data.

[1707] 6. Data Preprocessing

[1708] The server corrects misrecognition and formatting inconsistencies in character data extracted by OCR processing.

[1709] 7. Categorization

[1710] The server uses the preprocessed text data to classify expenses into categories such as food expenses, utility expenses, and medical expenses, using a machine learning algorithm.

[1711] 8. Monthly tally

[1712] The server aggregates the classified data by month and calculates the total expenditure amount and expenditure amount by category.

[1713] 9. Data Retention

[1714] The collected data is stored digitally so that users can check it later.

[1715] User behavior

[1716] 10. Data Verification

[1717] Users can check the aggregated spending data through the app, and can view monthly spending status and breakdowns by category.

[1718] 11. Receiving Advice

[1719] The user receives advice from the system. For example, if this month's food expenses have increased compared to last month, the app will suggest "reviewing your food expenses."

[1720] Specific examples

[1721] Example 1

[1722] A user goes shopping at a supermarket and receives a receipt.

[1723] The user launches the app and takes a photo of the receipt.

[1724] The device sends the captured image to the server.

[1725] The server analyzes the received image using OCR processing and extracts the text data.

[1726] The server classifies the extracted data as food expenses and reflects this in the monthly total amount.

[1727] Users can check their total food expenses through the app, which helps them manage their future spending.

[1728] Example 2

[1729] The user visits a hospital and receives a receipt.

[1730] The user uses the app to take a photo of the receipt.

[1731] The device sends the photo to the server.

[1732] The server analyzes the images and classifies them as medical expenses.

[1733] The server compiles medical expense data by month and organizes it as data for tax returns.

[1734] Users can check this month's medical expenses using the app, which can be useful when filing their tax returns.

[1735] In this way, the system of the present invention provides a means for efficiently managing receipts and visualizing expenditures.

[1736] The processing flow will be explained below.

[1737] Program processing

[1738] Step 1:

[1739] Subject: User

[1740] The user launches a receipt management app on their smartphone or tablet. The app has a photo capture function and a server communication function.

[1741] Step 2:

[1742] Subject: User

[1743] The user takes a photo of the receipt using the app's camera function, and the image is saved in the device's temporary storage.

[1744] Step 3:

[1745] Subject: Terminal

[1746] The device encrypts the captured image data, ensuring the security of the data.

[1747] Step 4:

[1748] Subject: Terminal

[1749] The device sends the encrypted image data to a server via the Internet using a secure communication protocol.

[1750] Step 5:

[1751] Subject: Server

[1752] The server receives the image data sent from the terminal and temporarily stores it. The received data is automatically added to the processing queue.

[1753] Step 6:

[1754] Subject: Server

[1755] The server performs optical character recognition (OCR) on the image data it receives, extracting the text information written on the receipt and saving it as text data.

[1756] Step 7:

[1757] Subject: Server

[1758] The server performs pre-processing to correct misrecognition and formatting inconsistencies in the character data extracted by OCR, thereby improving the accuracy of the data.

[1759] Step 8:

[1760] Subject: Server

[1761] The server classifies the preprocessed text data into specific categories (e.g., food expenses, utility bills, medical expenses) using machine learning algorithms.

[1762] Step 9:

[1763] Subject: Server

[1764] The server aggregates the categorized data by month and calculates the total expenditure for each category. The aggregated results are stored in a database and converted into a format that can be viewed by the user.

[1765] Step 10:

[1766] Subject: Server

[1767] The server organizes the results into the format required for tax returns and stores specific information such as medical expenses and donations separately.

[1768] Step 11:

[1769] Subject: User

[1770] Users can check the aggregated spending data through the app. Users can view monthly spending status and breakdowns by category.

[1771] Step 12:

[1772] Subject: User

[1773] The user receives advice from the system. For example, if food expenses have increased from last month, the user will receive a notification to "consider reviewing food expenses."

[1774] In this way, by performing specific processing for each step, the system efficiently manages receipts and makes expenditures visible.

[1775] Example 1

[1776] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1777] In recent years, receipt management has become increasingly complex, and there is a growing need to efficiently process, classify, and aggregate large volumes of receipts. Furthermore, to utilize receipt information for tax returns and expense management, it is essential to accurately and quickly extract data, correct misidentifications, and classify it into the necessary categories. Furthermore, to make it easier for users to manage their daily expenses, a system that allows easy data storage and viewing and provides appropriate advice is required.

[1778] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1779] In this invention, the server includes optical character recognition means for extracting character data from received images, means using a machine learning algorithm for classifying the extracted character data into specific categories, and preprocessing means for correcting misrecognition and formatting inconsistencies in the extracted character data. This allows for accurate extraction of text data from receipt images, classification into appropriate categories, and monthly aggregation. The system also includes means for organizing and saving information necessary for tax returns, thereby streamlining expense management and tax processing. Furthermore, a function for providing advice to users facilitates expense management.

[1780] The "means for taking a picture of a receipt" is a function that allows a user to take a picture of a receipt using a terminal.

[1781] The "means for transmitting the photographed image of the receipt to the server" is a function that enables the terminal to encrypt the photographed image data and transfer it to the server via the Internet.

[1782] "Optical character recognition means for extracting character data from received images" is a technology in which the server analyzes image data and extracts character information written on the receipt as text data.

[1783] "Means using a machine learning algorithm to classify extracted character data into specific categories" refers to an algorithm that the server uses machine learning technology to classify extracted character data into categories such as food expenses, utility expenses, and medical expenses.

[1784] The "means for aggregating classified data by month" is a function that allows the server to calculate the total monthly expenditure and the expenditure by category based on the data classified by category.

[1785] "Means for storing aggregated data and providing it to users" refers to the function that enables the server to safely store the aggregated data and allow users to view and check it later.

[1786] "Preprocessing means for correcting misrecognition or formatting inconsistencies in extracted character data" refers to a process in which the server corrects characters misrecognized during OCR processing or formatting inconsistencies to generate accurate text data.

[1787] The "means for providing advice to the user" is a function by which the server provides financial advice and suggestions for spending management based on the user's spending data.

[1788] A specific embodiment for implementing a receipt management system according to the present invention will be described. The purpose of this system is to efficiently manage receipts that users receive on a daily basis and to tally and classify expenditure data.

[1789] The system starts when a user takes a picture of a receipt using a device such as a smartphone or tablet. A receipt management app is installed on this device, which integrates a camera function and a server communication function. When a user launches the app and takes a picture of a receipt using the camera function, the image is saved in temporary storage on the device.

[1790] The device then encrypts the captured photo data and sends it over the Internet to a server using AES (Advanced Encryption Standard) encryption and HTTPS (Hyper Text Transfer Protocol Secure) to ensure secure communications.

[1791] The server receives the image data sent from the device and securely stores it in storage. The received data is automatically queued for processing. The server then analyzes the received image data using optical character recognition (OCR) technology to extract the text information listed on the receipt. Specifically, Google Cloud Vision API or other OCR software is used. The text data extracted by the OCR process may contain recognition errors or formatting inconsistencies, so the server performs preprocessing to correct these. This preprocessing includes spell checking and text formatting.

[1792] After preprocessing, the text data is then classified into specific categories using machine learning algorithms, such as libraries like scikit-learn and TensorFlow, including categories like food, utility, and medical expenses.

[1793] The classified data is aggregated monthly by the server, efficiently organized using an SQL database, and then securely stored in digital infrastructure such as Amazon S3 or Google Cloud Storage, where users can review it later.

[1794] Users can check the collected expenditure data through the app. When users press the "Check Data" button, monthly expenditure status and breakdown by category will be displayed. The system also has a function that allows users to receive financial advice. For example, if this month's food expenses have increased compared to last month, the system will suggest that users "review their food expenses."

[1795] Specific examples

[1796] Example 1

[1797] A user goes shopping at a supermarket and receives a receipt.

[1798] The user launches the app and takes a photo of the receipt.

[1799] The images captured by the device are encrypted using AES and sent to the server.

[1800] The server processes the images received using the Google Cloud Vision API and extracts text data.

[1801] The server classifies the extracted data as food expenses and reflects this in the monthly total amount.

[1802] The user checks the total food expenses through the app and receives the advice, "This month's food expenses have increased by 20% compared to last month."

[1803] Example 2

[1804] The user visits a hospital and receives a receipt.

[1805] The user uses the app to take a photo of the receipt.

[1806] The device encrypts the photo and sends it to the server.

[1807] The server analyzes the image using OCR processing and classifies it as medical expenses.

[1808] The server compiles medical expense data by month and organizes it as data for tax returns.

[1809] The user checks this month's medical expenses in the app and receives advice that "medical expense deductions may be applicable when filing your tax return."

[1810] Example prompts for generative AI models

[1811] "Describe in natural language the process of a receipt management system, from a user taking a photo of a receipt to a server parsing it to generate spending data. Explain each step of the process in detail and provide two concrete examples."

[1812] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1813] Step 1: Launch the app

[1814] A user launches a receipt management app on a smartphone or tablet. The app integrates a camera function and a server communication function. The input is the user's operation, and the output is the running status of the app.

[1815] process:

[1816] The user taps the app icon to launch it.

[1817] The app will launch and display the home screen.

[1818] Step 2: Take a photo of your receipt

[1819] The user takes a photo of the receipt using the camera function in the app. When the user presses the capture button, the camera takes a photo and the image is saved in temporary storage on the device. The input is the execution of the camera function, and the output is the captured image.

[1820] process:

[1821] The user presses the "Take Photo" button.

[1822] The device activates the camera, and the user frames the receipt and then presses the shutter button.

[1823] The image is saved to temporary storage.

[1824] Step 3: Sending image data

[1825] The device encrypts the captured photo data and sends it to a server over the Internet. This process uses AES encryption and the HTTPS protocol. The input is the image data, and the output is the encrypted data and its transmission.

[1826] process:

[1827] The device reads the image data from temporary storage.

[1828] Encrypt image data using AES.

[1829] The encrypted data is sent to the server using HTTPS.

[1830] Step 4: Receiving image data

[1831] The server receives image data sent from the device and stores it securely in storage. The received data is automatically registered in the processing queue. The input is encrypted image data, and the output is decrypted image data and its storage.

[1832] process:

[1833] The server receives the encrypted data using the HTTPS protocol.

[1834] The received data is decrypted using AES.

[1835] The decrypted data is stored in the server's storage and queued for processing.

[1836] Step 5: OCR processing

[1837] The server analyzes the received image data using optical character recognition (OCR) technology and extracts text information. Specifically, OCR software is used. The input is the decoded image data, and the output is the extracted text data.

[1838] process:

[1839] The server calls the Google Cloud Vision API or other OCR software to analyze the image data.

[1840] The text information written on the receipt is extracted using OCR processing.

[1841] The extracted text data is temporarily saved.

[1842] Step 6: Preprocessing the data

[1843] The server corrects misrecognition and formatting inconsistencies in the character data extracted by the OCR process. This process checks the spelling and formatting of the text data. The input is the extracted text data, and the output is the corrected text data.

[1844] process:

[1845] The server performs a spell check on the extracted text data.

[1846] Detect and correct formatting inconsistencies.

[1847] The corrected text data is saved.

[1848] Step 7: Categorize

[1849] The server classifies the preprocessed text data into categories using machine learning algorithms. The input is the corrected text data, and the output is the data classified into categories.

[1850] process:

[1851] The server invokes the machine learning model using libraries such as scikit-learn and TensorFlow.

[1852] The preprocessed data is fed into the model and categorical labels are assigned.

[1853] Store classified data.

[1854] Step 8: Monthly Tally

[1855] The server aggregates the classified data by month and calculates the total expenditure amount and expenditure amount by category. The input is data classified by category, and the output is the aggregated data.

[1856] process:

[1857] The server aggregates the categorized data by month using a database query.

[1858] Calculate total expenses and expenses by category for each month.

[1859] Save the summary results.

[1860] Step 9: Save your data

[1861] The server stores the aggregated data for users to review later. The input is the aggregated data, and the output is the stored data.

[1862] process:

[1863] The server stores the data on a digital platform such as Amazon S3 or Google Cloud Storage.

[1864] Verify that the saved data is stored correctly.

[1865] Step 10: Data verification

[1866] The user checks the aggregated spending data through the app. The input is the user request, and the output is the displayed spending data.

[1867] process:

[1868] The user presses the "Check Data" button in the app.

[1869] The server sends the stored aggregated data to the app in response to a user request.

[1870] The device displays the received data, allowing the user to check monthly spending and breakdowns by category.

[1871] Step 11: Getting advice

[1872] The user receives advice from the system: the input is expenditure data and the output is an advice message.

[1873] process:

[1874] The server analyzes spending data to detect unusual patterns and areas for improvement.

[1875] An advice message is generated and notified to the user.

[1876] Users can review advice messages to help them manage their future spending.

[1877] (Application example 1)

[1878] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1879] Traditional receipt management systems require the manual management of paper receipts, which is time-consuming and labor-intensive. They also lack the ability to efficiently import electronic receipts, making overall spending management cumbersome. They also make it difficult for users to understand their spending trends and identify areas for improvement. Furthermore, extracted character data can be misrecognized or incompletely classified, making accurate spending management difficult.

[1880] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1881] In this invention, the server includes a means for transmitting an image of a receipt to the server, an optical character recognition means for extracting character data from the received image, a means for classifying the extracted character data into specific categories, a means for automatically capturing electronic receipts and transmitting them to the system, and a means for providing the user with spending advice based on the compiled data. This allows users to efficiently manage not only paper receipts but also electronic receipts, making it easy to understand spending trends and areas for improvement. Furthermore, accurate classification of the extracted character data makes user spending management more accurate and efficient.

[1882] "Means for taking an image of a receipt" refers to a mobile device equipped with a camera function for capturing a digital image of a paper receipt.

[1883] "Means for transmitting the photographed image of the receipt to the server" refers to a communication function for transferring the digital image data of the acquired receipt to the server via a network.

[1884] "Optical character recognition means" means optical character recognition (OCR) technology for identifying characters from a digital image and extracting them as text data.

[1885] "Means for classifying extracted character data into specific categories" refers to algorithms or machine learning models that automatically organize and classify text data obtained through OCR into specific categories, such as food expenses and medical expenses.

[1886] "Means for aggregating classified data by month" refers to a program that aggregates data classified into specific categories on a monthly basis and calculates total expenditures and expenditures by category.

[1887] "Means for storing aggregated data and providing it to users" refers to the interface that stores monthly aggregated data in the server and allows users to check and view that data.

[1888] "Means for automatically receiving and transmitting electronic receipts to the system" refers to a function for automatically receiving electronic receipts generated from electronic payment services and managing them on the system.

[1889] "Means for providing spending advice to users based on aggregated data" refers to functionality that analyzes aggregated data and makes suggestions for improving or managing a user's spending habits.

[1890] This invention begins when a user takes a photo of a receipt using a device such as a smartphone and sends the data to a server. The server then analyzes the received image data using optical character recognition (OCR) technology, classifies the extracted text data by category, and aggregates it by month. This system allows users to efficiently manage their expenses. It also has a function that links with electronic payment services to automatically import electronic receipts and send them to the system.

[1891] Hardware and Software

[1892] Hardware:

[1893] Smartphone or tablet camera: Used to take a picture of the receipt.

[1894] Server: Used to receive, analyze, and store image data.

[1895] software:

[1896] OpenCV: Used for image preprocessing (grayscale conversion and binarization).

[1897] pytesseract: A library for OCR processing, used to extract text data from images.

[1898] scikit-learn: Used to classify character data extracted by machine learning algorithms into categories.

[1899] Program processing explanation

[1900] The server first receives the image data sent by the user and preprocesses the image using OpenCV. Next, it uses pytesseract to extract text data from the preprocessed image. It then uses the scikit-learn machine learning algorithm to classify the extracted text data into specific categories. The classified data is aggregated by month, and the total expenditure amount and expenditure amount by category are calculated. The results are stored on the server and made available for users to view and check through the application.

[1901] The company has also introduced a system for automatically capturing digital receipts, allowing users to manage both paper and digital receipts in one place, and it also has a function that provides users with spending advice based on the aggregated data.

[1902] Examples and prompts

[1903] Example 1

[1904] Let's say a user goes shopping at a supermarket and receives a receipt. When the user launches the smartphone app and takes a photo of the receipt, the device sends the image to the server. The server analyzes the received image using OCR processing and extracts text data. The system then classifies the extracted data as food expenses and reflects it in the monthly total. The user can check the total food expenses through the app, which can be used to manage future expenses.

[1905] Example 2

[1906] When a user receives a medical receipt from a hospital, they use the app to take a photo of the receipt, and the device sends the photo to the server. The server analyzes the image and classifies it as medical expenses. This data is aggregated monthly and organized as tax return data. Users can check their medical expenses for the month in the app, which can be useful when filing their tax return.

[1907] Prompt Sentence Examples

[1908] "A user's food spending this month has increased by 20% compared to last month. Please provide advice on how to address this increase."

[1909] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1910] Step 1:

[1911] The user launches the application and takes a picture of the receipt. When the user takes a picture of a paper receipt using a smartphone or tablet, the image is temporarily saved on the device. The input is the image of the receipt, and the output is the temporarily saved digital image data.

[1912] Step 2:

[1913] The device sends the captured image data to the server. The device transfers the encrypted image data to the server via the Internet. The input is the temporarily stored digital image data, and the output is the image data sent to the server.

[1914] Step 3:

[1915] The server stores the received image data and performs preprocessing for OCR. The server uses OpenCV to convert the image to grayscale and perform binarization. The input is the image data sent to the server, and the output is the preprocessed image data.

[1916] Step 4:

[1917] The server performs OCR processing on the preprocessed image and extracts character data. The server extracts the text data using pytesseract and saves it as character data in UTF-8 format. The input is the preprocessed image data, and the output is the extracted character data.

[1918] Step 5:

[1919] The server preprocesses the extracted character data to check for misrecognitions and inconsistencies. The server uses regular expressions to clean the extracted data and correct misrecognitions. The input is the extracted character data, and the output is the preprocessed, clean character data.

[1920] Step 6:

[1921] The server classifies the preprocessed text data into specific categories. The server uses scikit-learn's machine learning algorithm to automatically classify the data into categories such as food expenses and medical expenses. The input is the preprocessed, clean text data, and the output is the classified category data.

[1922] Step 7:

[1923] The server aggregates the classified data by month and calculates the total expenditure amount and expenditure amount by category. The server uses an SQL database to store and aggregate the data and organize it by month. The input is the classified category data, and the output is the expenditure data aggregated by month.

[1924] Step 8:

[1925] The server stores the aggregated data and provides it to the user. The server stores the aggregated results in a database and allows the user to view the data through the app. The input is monthly aggregated expenditure data, and the output is the stored aggregated data and the data displayed in the user interface.

[1926] Step 9:

[1927] The server provides spending advice to the user based on the aggregated data. The server uses a generative AI model to analyze the user's spending data and generate advice based on prompts. The input is monthly aggregated spending data and prompts, and the output is the generated advice.

[1928] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1929] A specific embodiment of the receipt management system combined with an emotion engine according to the present invention will now be described.

[1930] The system begins when a user takes a photo of a receipt using a device such as a smartphone or tablet and sends the image to a server. The server analyzes the received image data and extracts the necessary information using optical character recognition (OCR). The extracted text data is classified by category, and monthly expenditures are calculated based on this. The system then uses an emotion engine to recognize the user's emotions and customize spending advice based on this.

[1931] Program processing

[1932] Terminal side processing

[1933] 1. Launch the app

[1934] A user launches a receipt management app that integrates photo capture, emotion recognition, and server communication functions.

[1935] 2. Take a photo of your receipt

[1936] The user takes a photo of the receipt using the app's camera function, and the image is saved in the device's temporary storage.

[1937] 3. Sending image data

[1938] The device encrypts the photo data and sends it to a server over the Internet, ensuring secure communication.

[1939] Server-side processing

[1940] 4. Receiving image data

[1941] The server receives the image data sent from the terminal and temporarily stores it. The received data is automatically added to the processing queue.

[1942] 5. OCR Processing

[1943] The server performs optical character recognition (OCR) on the image data, extracting the text information written on the receipt and saving it as text data.

[1944] 6. Data Preprocessing

[1945] The server performs pre-processing to correct misrecognition and formatting inconsistencies in the character data extracted by OCR, thereby improving the accuracy of the data.

[1946] 7. Categorization

[1947] The server classifies the preprocessed text data into specific categories (e.g., food expenses, utility bills, medical expenses) using machine learning algorithms.

[1948] 8. Monthly tally

[1949] The server aggregates the categorized data by month and calculates the total expenditure for each category. The aggregated results are stored in a database and converted into a format that can be viewed by the user.

[1950] 9. Emotion recognition

[1951] The server uses an emotion engine to recognize the user's emotions from the user's input data and interaction logs, for example, by analyzing facial expressions and usage patterns displayed while the user is using the app.

[1952] 10. Customizing Advice

[1953] The server then customizes spending advice based on the perceived emotion, for example, gently suggesting savings advice if the user is feeling stressed.

[1954] 11. Data Retention

[1955] The aggregated data and customized advice will be stored digitally for users to review at a later date.

[1956] User behavior

[1957] 12. Data Verification

[1958] Users can check the aggregated spending data through the app. Users can view monthly spending status and breakdowns by category.

[1959] 13. Receiving Advice

[1960] The user receives customized advice from the system, for example, if their food expenses have increased since last month, the system will display advice tailored to their current emotional state.

[1961] Specific examples

[1962] Example 1

[1963] A user goes shopping at a supermarket and receives a receipt.

[1964] The user launches the app and takes a photo of the receipt.

[1965] The device sends the captured image to the server.

[1966] The server analyzes the received image using OCR processing and extracts the text data.

[1967] The server classifies the extracted data as food expenses and reflects this in the monthly total amount.

[1968] The server recognizes the user's emotions and makes gentle suggestions to save money if stress levels are high.

[1969] Users can check the total food cost and advice through the app to help manage their future spending.

[1970] Example 2

[1971] The user visits a hospital and receives a receipt.

[1972] The user uses the app to take a photo of the receipt.

[1973] The device sends the photo to the server.

[1974] The server analyzes the images and classifies them as medical expenses.

[1975] The server compiles medical expense data by month and organizes it as data for tax returns.

[1976] The server analyzes the user's emotions and, if anxiety is high, provides reassuring advice on reviewing spending, for example.

[1977] Users can check this month's medical expenses using the app, which can be useful when filing their tax returns.

[1978] In this way, the system of the present invention efficiently manages receipts and visualizes expenditures, and by combining it with an emotion engine, it provides expenditure advice tailored to the user's emotional state.

[1979] The processing flow will be explained below.

[1980] Program processing

[1981] Step 1:

[1982] Subject: User

[1983] A user launches a receipt management app on a smartphone or tablet, which is equipped with a camera, emotion recognition, and server communication functions.

[1984] Step 2:

[1985] Subject: User

[1986] The user takes a photo of the receipt using the app's camera function, and the image is saved in the device's temporary storage.

[1987] Step 3:

[1988] Subject: Terminal

[1989] The device encrypts the captured image data, ensuring the security of the data.

[1990] Step 4:

[1991] Subject: Terminal

[1992] The device sends the encrypted image data to a server via the Internet using a secure communication protocol.

[1993] Step 5:

[1994] Subject: Server

[1995] The server receives the image data sent from the terminal and temporarily stores it. The received data is automatically added to the processing queue.

[1996] Step 6:

[1997] Subject: Server

[1998] The server performs optical character recognition (OCR) on the image data it receives, extracting the text information written on the receipt and saving it as text data.

[1999] Step 7:

[2000] Subject: Server

[2001] The server performs pre-processing to correct misrecognition and formatting inconsistencies in the character data extracted by OCR, thereby improving the accuracy of the data.

[2002] Step 8:

[2003] Subject: Server

[2004] The server classifies the preprocessed text data into specific categories (e.g., food expenses, utility bills, medical expenses) using machine learning algorithms.

[2005] Step 9:

[2006] Subject: Server

[2007] The server aggregates the categorized data by month and calculates the total expenditure for each category. The aggregated results are stored in a database and converted into a format that can be viewed by the user.

[2008] Step 10:

[2009] Subject: Server

[2010] The server uses an emotion engine to recognize the user's emotions from the user's input data and interaction logs, for example, by analyzing facial expressions and usage patterns displayed while the user is using the app.

[2011] Step 11:

[2012] Subject: Server

[2013] The server then customizes spending advice based on the perceived emotion, for example, gently suggesting savings advice if the user is feeling stressed.

[2014] Step 12:

[2015] Subject: Server

[2016] The server then stores the results and customized advice digitally, allowing users to review them later.

[2017] Step 13:

[2018] Subject: User

[2019] Users can check the aggregated spending data through the app. Users can view monthly spending status and breakdowns by category.

[2020] Step 14:

[2021] Subject: User

[2022] The user receives customized advice from the system, for example, if their food expenses have increased since last month, the system will display advice tailored to their current emotional state.

[2023] Example 2

[2024] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2025] In modern society, managing and organizing the large number of receipts generated in daily life and business situations is a significant burden for users. Taking images of receipts, extracting information, and accurately categorizing and aggregating them is particularly time-consuming and labor-intensive. Furthermore, since appropriate spending advice is not provided based on the user's emotional state, it can easily lead to stressful situations. Conventional technologies have not been able to adequately resolve these issues.

[2026] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2027] In this invention, the server includes means for capturing an image of a receipt, means for transmitting the captured image of the receipt to the server, optical character recognition means for extracting character data from the received image, means for classifying the extracted character data into specific categories, means for aggregating the classified data by month, means for recognizing a user's emotion, means for customizing advice on spending based on the recognized emotion, and means for saving the aggregated data and customized advice and providing it to the user. This not only improves the efficiency of receipt management work but also enables spending advice tailored to the user's emotional state.

[2028] A "receipt" is a document issued to prove the content and amount of a transaction when purchasing a product or using a service.

[2029] An "image" is visual information recorded in digital form, and is data that includes text, graphics, etc.

[2030] A "server" is a computer system that provides data and services to other devices (terminals) within a network.

[2031] A "terminal" is a communication device or computer that can only function when connected to a network.

[2032] Optical character recognition (OCR) is a technology that reads character information in an image and converts it into digital text.

[2033] A "category" is a division for classifying data according to its type or characteristics.

[2034] "Aggregation" is the process of aggregating multiple data sets and calculating statistical information such as totals and averages.

[2035] "Emotion recognition" is a technology that analyzes and determines a user's emotional state from data.

[2036] "Advice" is advice or suggestions offered to encourage optimal action or decision in a particular situation.

[2037] "Storage" is the act of writing down data on a recording medium for later use.

[2038] "User" refers to an individual or corporation that uses this system to manage receipts and check data.

[2039] This section describes a specific example of combining an emotion engine with a receipt management system according to the present invention. This system begins when a user takes an image of a receipt using a device such as a smartphone or tablet and sends the image to a server. The specific hardware used is a smartphone or tablet, and the software used is a dedicated receipt management app.

[2040] The server-side processing includes receiving image data, OCR processing, data preprocessing, categorization, monthly aggregation, emotion recognition, and generating and providing customized advice. These processes are performed using the following technologies:

[2041] Server-side processing

[2042] 1. Receiving image data

[2043] The server receives the image data sent from the device and temporarily stores it. The HTTPS protocol is used for communication, and the AES encryption algorithm ensures data security.

[2044] 2. OCR processing

[2045] The server uses optical character recognition technology, such as Google Cloud Vision API, to extract text from the image data. This process converts the text data on the receipt into text format.

[2046] 3. Data Preprocessing

[2047] The character data extracted by OCR may contain misrecognition errors or formatting inconsistencies, and the server uses a Python-based data cleaning tool to correct these errors.

[2048] 4. Categorization

[2049] A machine learning algorithm (e.g., the BERT model) is used to classify the preprocessed text data into specific categories (e.g., food expenses, utility expenses, medical expenses), thereby achieving accurate data classification.

[2050] 5. Monthly tally

[2051] The categorized data is aggregated by month, and the total expenditure for each category is calculated. The aggregated results are stored in a database (e.g., MySQL) and can be viewed by users in HTML or JSON format.

[2052] 6. Emotion recognition

[2053] We use Microsoft Azure's Emotion API to recognize the user's emotional state from the user's input data and interaction logs, which allows us to accurately determine the user's emotional state (e.g., stress, relief).

[2054] 7. Customizing Advice

[2055] Based on the recognized emotions, spending advice is customized using a generative AI model (e.g., OpenAI's GPT-4). For example, it generates "food expenses are increasing, but reasonable savings suggestions are generated due to high stress."

[2056] 8. Data storage and provision

[2057] The aggregated data and customized advice is stored on Amazon S3 or Google Cloud Storage, which users can then review later through the app.

[2058] User behavior

[2059] Users can view aggregated spending data through the app. They can view monthly spending status and category breakdowns in graphs and tables for specific spending management. They can also receive customized spending advice, allowing them to review their spending appropriately based on their emotional state.

[2060] Specific examples

[2061] As a concrete example, let's consider the process of a user shopping at a supermarket, receiving a receipt, and then taking a photo of the receipt using the app. The device sends this photo data to a server, which then analyzes the image using OCR to extract text data. The extracted data is classified as "food expenses" and reflected in the monthly total. If the user's stress level is high using emotion recognition, gentle suggestions for saving money are made, which the user can confirm through the app and use to help manage future expenses.

[2062] Next, when the user visits a hospital for a medical examination and receives a receipt, the same procedure is followed. The app takes a photo of the receipt, and the server classifies it as a medical expense. The medical expense data is aggregated by month and organized for tax return filing. Furthermore, if the user is highly anxious based on sentiment analysis, the app provides advice on reviewing expenses to reassure them.

[2063] Example prompt using a generative AI model: "My food expenses have been increasing recently, and I'm under a lot of stress. Can you tell me some reasonable ways to save money?"

[2064] The above is a specific embodiment for implementing the present invention. This system efficiently manages receipts and visualizes expenditures, and by combining it with an emotion engine, it provides expenditure advice tailored to the user's emotional state.

[2065] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2066] Step 1:

[2067] Launching the app

[2068] The user launches the receipt management app on their smartphone or tablet. The app prompts the user for authentication information (e.g., ID and password) the first time it is launched. If authentication is successful, the user is redirected to the main screen.

[2069] Input: Launching an app through user interaction

[2070] Output: The main screen of the app is displayed.

[2071] Step 2:

[2072] Taking a photo of the receipt

[2073] The user takes a photo of the receipt using the app's camera, and the image is saved in temporary device storage in JPEG format, ready to be sent to the server later.

[2074] Input: User-taken image of receipt

[2075] Output: JPEG image data is saved to temporary storage

[2076] Step 3:

[2077] Sending image data

[2078] The device encrypts the captured image data using the 128-bit AES encryption algorithm, then sends the encrypted image data to the server using HTTPS, with the TLS protocol ensuring communication security.

[2079] Input: JPEG format image data

[2080] Output: The encrypted image data is sent to the server.

[2081] Step 4:

[2082] Receiving image data

[2083] The server receives the image data sent from the device and temporarily stores it. The received data is automatically added to the processing queue and awaits analysis.

[2084] Input: Encrypted image data

[2085] Output: Temporarily saved image data

[2086] Step 5:

[2087] OCR processing

[2088] The server uses the Google Cloud Vision API to perform OCR processing on the image data, extracting the text information from the receipt and saving it as text data.

[2089] Input: Temporarily saved image data

[2090] Output: Extracted text data

[2091] Step 6:

[2092] Data Preprocessing

[2093] The server performs pre-processing to correct misrecognition and formatting inconsistencies in the character data extracted by OCR. For example, if the number "1" is misrecognized as a "lowercase l" or an "uppercase I," it will be converted to the correct format.

[2094] Input: Extracted text data

[2095] Output: Preprocessed text data

[2096] Step 7:

[2097] Categorization

[2098] The server classifies the preprocessed text data into specific categories (e.g., food expenses, utility expenses, medical expenses) using a machine learning algorithm (e.g., the BERT model).

[2099] Input: Preprocessed text data

[2100] Output: Categorized data

[2101] Step 8:

[2102] Monthly tally

[2103] The server aggregates the categorized data by month and calculates the total expenditure for each category. The aggregated results are stored in a database (e.g., MySQL) and can be viewed by users in HTML or JSON format.

[2104] Input: Categorical data

[2105] Output: Monthly aggregated data

[2106] Step 9:

[2107] emotion recognition

[2108] The server uses Microsoft Azure's Emotion API to recognize the user's emotional state from the user's input data and interaction logs, thereby accurately determining the user's emotional state (e.g., stress, relief).

[2109] Input: User input data and interaction logs

[2110] Output: Recognized emotion data

[2111] Step 10:

[2112] Customizing Advice

[2113] The server then uses a generative AI model (e.g., OpenAI's GPT-4) to customize spending advice based on the recognized emotions. For example, it generates a reasonable saving suggestion, saying, "Food expenses are increasing, but stress is high."

[2114] Input: Recognized emotion data

[2115] Output: Customized advice

[2116] Step 11:

[2117] Data storage and provision

[2118] The server stores the aggregated data and customized advice on Amazon S3 or Google Cloud Storage, which users can then view via the app.

[2119] Input: Monthly aggregated data and customized advice

[2120] Output: Stored data and advice

[2121] Step 12:

[2122] Data confirmation

[2123] The user accesses the "Data Check" section of the app, where they can view their monthly spending and a breakdown by category in graphs and tables.

[2124] Input: Data confirmation request by user operation

[2125] Output: Screen showing expenditure status and breakdown

[2126] Step 13:

[2127] Receiving advice

[2128] The user accesses the "Advice" section and sees customized spending advice from the system. For example, because food expenses have increased since last month, soft suggestions are displayed that are tailored to the user's emotional state.

[2129] Input: User initiated advice request

[2130] Output: Screen showing customized advice

[2131] (Application example 2)

[2132] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2133] In today's brick-and-mortar shopping environment, customers lack the means to efficiently manage receipts and understand their spending in real time. Furthermore, when customers experience emotional stress related to their spending habits, they often lack the ability to receive appropriate advice and support. There is a need for a system that can solve these problems and provide more personalized spending management and psychological support to customers.

[2134] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2135] In this invention, the server includes means for capturing an image of a receipt, means for transmitting the captured image of the receipt to the server, optical character recognition means for extracting character data from the received image, means for classifying the extracted character data into specific categories, means for aggregating the classified data by month, means for storing and providing the aggregated data to a user, and means for recognizing the user's emotions and customizing spending advice based on the emotion data, thereby enabling customers to manage their spending in real time while shopping at a physical store and receive optimal spending advice based on their emotions.

[2136] The "means for taking a picture of a receipt" is a function that allows a user to take a picture of a receipt for a product purchased at a physical store using the camera on a mobile device such as a smartphone.

[2137] The "means for transmitting the photographed image of the receipt to the server" is a communication function that encrypts the photographed image of the receipt and transmits it securely to the server.

[2138] "Optical character recognition means for extracting character data from received images" refers to a function that uses optical character recognition (OCR) technology to analyze character information from receipt images received by the server and extract it as text data.

[2139] The "means for classifying extracted character data into specific categories" is a function for classifying character data extracted by OCR into specific categories such as food, daily necessities, and medical expenses.

[2140] The "means for aggregating classified data by month" is a function for aggregating expenditure data classified by category by month.

[2141] The "means for storing aggregated data and providing it to the user" is a function for storing the aggregated results in a database and displaying them so that the user can check them through the app.

[2142] "Means for recognizing user emotions and customizing spending advice based on this emotional data" refers to a function that analyzes user interactions and input data to recognize emotions and provides individually optimized spending advice based on the results of this recognition.

[2143] This invention is a system that allows customers in physical stores to use their smartphones to manage receipts and visualize spending in real time, and provides personalized advice based on their emotions. Specific embodiments of the invention are described below.

[2144] System Overview

[2145] It consists of three elements: a server, a terminal (such as a smartphone), and a user. Details of each element are as follows:

[2146] Required Hardware and Software

[2147] Hardware: Smartphone (compatible with Android and iOS), store Wi-Fi network, cloud server

[2148] Software: smartphone applications, OCR libraries (e.g., Tesseract), emotion recognition algorithms (e.g., Microsoft Azure Cognitive Services or Google Cloud Vision API)

[2149] Processing flow and data processing

[2150] 1. Terminal processing

[2151] App launch: A user enters a store and launches the app, which has integrated photo-taking and emotion recognition functions.

[2152] Taking a photo of a receipt: The user uses the camera function within the app to take a photo of the receipt. The captured image is saved in the device's temporary storage.

[2153] Transmission of image data: The device encrypts the image data and transmits it to a server via the Internet.

[2154] 2. Server-side processing

[2155] Receiving image data: The server receives the image data sent from the terminal and temporarily stores it.

[2156] OCR processing: The server performs OCR processing on the image data received and extracts the text information written on the receipt.

[2157] Data preprocessing: Preprocessing is performed to correct misrecognition and formatting inconsistencies in the character data extracted by OCR processing.

[2158] Category Classification: Classify the preprocessed text data into specific categories such as food, daily necessities, medical expenses, etc., using machine learning algorithms (scikit-learn, TensorFlow).

[2159] Monthly aggregation: The classified data is aggregated by month to calculate the total expenditure amount for each category.

[2160] Emotion Recognition: Analyzes user interactions and input data and recognizes user emotions with the emotion engine. Emotion recognition is performed using Microsoft Azure Cognitive Services or Google Cloud Vision API.

[2161] Customized Advice: Customize spending advice based on recognized emotions. Customized advice is generated using recommendation algorithms.

[2162] 3. User Actions

[2163] Data confirmation: Users can check the aggregated expenditure data through the app. Users can view monthly expenditure status and breakdown by category.

[2164] Get advice: Users receive personalized spending advice based on their emotions.

[2165] Specific examples

[2166] Example 1

[2167] The user goes shopping at the grocery store and takes a photo of the receipt they receive at the register. The app uses OCR to classify it as a "food expense" and, after analyzing their past purchase history and emotional data, displays advice such as, "Your food expenses have increased compared to last month, so why not take advantage of the weekend sales?"

[2168] Example prompts for generative AI models

[2169] After a user shops at a physical store, they take a photo of the receipt with the app. The app then performs OCR processing to analyze the receipt's contents. The OCR-extracted text data is then categorized and aggregated by month. The app also analyzes the user's emotions and provides personalized spending advice based on their emotions.

[2170] In this way, the system of the present invention enhances the shopping experience in brick-and-mortar stores and gives customers peace of mind in managing their spending.

[2171] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2172] Step 1:

[2173] Launching the app

[2174] A user enters a physical store and launches the app on their smartphone, which has integrated photo-taking and emotion recognition functions.

[2175] Input: None

[2176] Output: App launch

[2177] Step 2:

[2178] Taking a photo of the receipt

[2179] A user purchases an item at a store and takes a photo of the receipt they receive at the cash register using the app's camera function.

[2180] Input: Receipt

[2181] Output: Receipt image data

[2182] Step 3:

[2183] Sending image data

[2184] The device encrypts the image of the receipt it captures and sends it to a cloud server via the Internet.

[2185] Input: Receipt image data

[2186] Output: Image data sent to the server

[2187] Step 4:

[2188] Receiving image data

[2189] The server receives the image data sent from the terminal and temporarily stores it.

[2190] Input: Image data sent

[2191] Output: Image data saved in temporary storage

[2192] Step 5:

[2193] OCR processing

[2194] The server performs OCR processing on the image data received and extracts the text information written on the receipt. An OCR library such as Tesseract is used.

[2195] Input: Image data

[2196] Output: Extracted character data

[2197] Step 6:

[2198] Data Preprocessing

[2199] The server corrects misrecognition and formatting inconsistencies in the text data extracted by the OCR process, for example, correcting misrecognized letters and numbers and arranging them into a consistent format.

[2200] Input: Extracted character data

[2201] Output: Preprocessed character data

[2202] Step 7:

[2203] Categorization

[2204] The server classifies the preprocessed text data into specific categories such as food, daily necessities, medical expenses, etc. It uses machine learning algorithms such as scikit-learn and TensorFlow.

[2205] Input: Preprocessed character data

[2206] Output: Categorized data

[2207] Step 8:

[2208] Monthly tally

[2209] The server aggregates the classified data by month and performs data calculations to calculate the total expenditure amount for each category.

[2210] Input: Categorical data

[2211] Output: Monthly aggregated data

[2212] Step 9:

[2213] emotion recognition

[2214] The server analyzes the user's interactions and input data while using the app, and performs data calculations to recognize the user's emotions using an emotion engine (such as Microsoft Azure Cognitive Services or Google Cloud Vision API). It analyzes the user's facial expressions and vocabulary patterns.

[2215] Input: Interaction data and input data

[2216] Output: Emotion recognition result

[2217] Step 10:

[2218] Customizing Advice

[2219] The server generates customized spending advice based on the perceived emotions, using recommendation algorithms such as collaborative filtering.

[2220] Input: Emotion recognition results and monthly aggregated data

[2221] Output: Customized spending advice

[2222] Step 11:

[2223] Data confirmation

[2224] Users can access aggregated spending data and personalized advice through the app. Users can view monthly spending and a breakdown by category.

[2225] Input: Monthly aggregated data and customized advice

[2226] Output: Spending data and advice displayed in a user interface

[2227] In this way, the system helps users manage their spending through a series of steps and provides optimal emotional advice.

[2228] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2229] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2230] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2231] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2232] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2233] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2234] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2235] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2236] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2237] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2238] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2239] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2240] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2241] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2242] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2243] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2244] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2245] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2246] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2247] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2248] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2249] The following is further disclosed regarding the above embodiment.

[2250] (Claim 1)

[2251] means for taking an image of the receipt;

[2252] means for transmitting the captured image of the receipt to a server;

[2253] optical character recognition means for extracting character data from the received image;

[2254] A means for classifying the extracted character data into specific categories;

[2255] A means of aggregating the classified data by month;

[2256] a means for storing and providing aggregated data to the user;

[2257] A system including:

[2258] (Claim 2)

[2259] 10. The system of claim 1, further comprising means for organizing and storing information required for filing tax returns.

[2260] (Claim 3)

[2261] 10. The system of claim 1, further comprising a preprocessing means for correcting misrecognition of the extracted character data.

[2262] "Example 1"

[2263] (Claim 1)

[2264] means for taking an image of the receipt;

[2265] means for transmitting the captured image of the receipt to a server;

[2266] optical character recognition means for extracting character data from the received image;

[2267] A means using machine learning algorithms to classify the extracted character data into specific categories; and

[2268] A means of aggregating the classified data by month;

[2269] a means for storing and providing aggregated data to the user;

[2270] a preprocessing means for correcting misrecognition and formatting inconsistencies of the extracted character data;

[2271] means for providing advice to a user;

[2272] A system including:

[2273] (Claim 2)

[2274] 10. The system of claim 1, further comprising means for organizing and storing information required for filing tax returns.

[2275] (Claim 3)

[2276] 10. The system of claim 1, further comprising means for securely storing the classified data.

[2277] "Application Example 1"

[2278] (Claim 1)

[2279] means for taking an image of the receipt;

[2280] means for transmitting the captured image of the receipt to a server;

[2281] optical character recognition means for extracting character data from the received image;

[2282] A means for classifying the extracted character data into specific categories;

[2283] A means of aggregating the classified data by month;

[2284] a means for storing and providing aggregated data to the user;

[2285] a means for automatically capturing and transmitting electronic receipts to the system;

[2286] means for providing spending advice to users based on the aggregated data;

[2287] A system including:

[2288] (Claim 2)

[2289] 10. The system of claim 1, further comprising means for organizing and storing information required for filing tax returns.

[2290] (Claim 3)

[2291] 10. The system of claim 1, further comprising a preprocessing means for correcting misrecognition of the extracted character data.

[2292] "Example 2: Combining Emotion Engines"

[2293] (Claim 1)

[2294] means for taking an image of the receipt;

[2295] means for transmitting the captured image of the receipt to a server;

[2296] optical character recognition means for extracting character data from the received image;

[2297] A means for classifying the extracted character data into specific categories;

[2298] A means of aggregating the classified data by month;

[2299] means for recognizing a user's emotion;

[2300] a means of customizing spending advice based on perceived emotions;

[2301] means for storing and providing aggregated data and customized advice to users;

[2302] A system including:

[2303] (Claim 2)

[2304] 10. The system of claim 1, further comprising means for organizing and storing information required for filing tax returns.

[2305] (Claim 3)

[2306] 10. The system of claim 1, further comprising a preprocessing means for correcting misrecognition of the extracted character data.

[2307] "Application example 2 when combining emotion engines"

[2308] (Claim 1)

[2309] means for taking an image of the receipt;

[2310] means for transmitting the captured image of the receipt to a server;

[2311] optical character recognition means for extracting character data from the received image;

[2312] A means for classifying the extracted character data into specific categories;

[2313] A means of aggregating the classified data by month;

[2314] a means for storing and providing aggregated data to the user;

[2315] means for recognizing user emotions and customizing spending advice based on this emotion data;

[2316] A system including:

[2317] (Claim 2)

[2318] 10. The system of claim 1, further comprising means for organizing and storing information required for filing tax returns.

[2319] (Claim 3)

[2320] 10. The system of claim 1, further comprising a preprocessing means for correcting misrecognition of the extracted character data. [Explanation of symbols]

[2321] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for taking an image of the receipt; means for transmitting the captured image of the receipt to a server; optical character recognition means for extracting character data from the received image; A means for classifying the extracted character data into specific categories; A means of aggregating the classified data by month; a means for storing and providing aggregated data to the user; A system including:

2. 10. The system of claim 1, further comprising means for organizing and storing information required for filing a tax return.

3. 2. The system of claim 1, further comprising preprocessing means for correcting misrecognition of extracted character data.

Images