system

Abstract

A system is provided.SOLUTION: A character information extraction unit that extracts character information from the receipt image using an OCR technique, an analysis unit that analyzes the character information extracted by the character information extraction unit and generates a store name, a product name, a price, and a purchase date and time, and a classification unit that classifies the information generated by the analysis unit for each category; The system includes a database registration means for storing information in a household account database, a display means for displaying the information stored in the household account database, and a push notification means for collating the household account database with an advertisement database and providing highly relevant bargain information by push notification.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, managing one's household finances is an important issue. However, traditional methods of managing household finances are time-consuming and difficult to maintain. Furthermore, with the spread of electronic payments, various payment methods exist, including not only cash but also credit cards and electronic money, making it difficult to centrally manage the details of these payments. Furthermore, analyzing spending patterns and providing information on deals are necessary for users to efficiently and effectively save money and manage their budgets, but this is difficult to achieve with traditional applications and methods. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems with a system that includes a photographing device for a user to photograph a receipt, a receiving device for receiving the receipt image photographed by the photographing device, a character information extraction device for extracting character information from the receipt image using OCR technology, an analysis device for analyzing the character information extracted by the character information extraction device and generating store name, product name, price, and purchase date and time, a database registration device for categorizing the information generated by the analysis device and storing it in a household accounting database, a display device for displaying the information stored in the household accounting database, and a push notification device for comparing the household accounting database with an advertising database and providing relevant discount information via push notification. This allows users to easily manage their household finances in detail, enabling efficient spending management and savings. Furthermore, by providing discount information based on spending patterns, the system can further support users' lifestyles.

[0006] "Photographing means" refers to a function or device that allows the user to photograph a receipt.

[0007] The "receiving means" is a function or system that receives the receipt image captured by the image capturing means.

[0008] "Text information extraction means" refers to a function or system that uses OCR technology to extract text information from receipt images.

[0009] The "analysis means" is a function or system that analyzes the character information extracted by the character information extraction means and generates the store name, product name, price, and purchase date and time.

[0010] The "database registration means" is a function or system that classifies the information generated by the analysis means into categories and stores the information in the household account book database.

[0011] The "display means" is a function or system for displaying information stored in the household accounting database to the user.

[0012] The "push notification means" is a function or system that compares the household accounting database with the advertising database and provides relevant discount information to the user via push notification.

[0013] The "expenditure forecasting means" is a function or system that forecasts future expenditures based on information generated by the analysis means and supports budget management.

[0014] The "history analysis means" is a function or system that analyzes the user's purchase history accumulated in the household account book database and calculates the purchase frequency of a particular store or product. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] MODE FOR CARRYING OUT THE INVENTION

[0037] The present invention provides a system that automatically generates a household account book by allowing a user to simply take a photo of a receipt. The following is a specific example for implementing the present invention.

[0038] System Overview

[0039] The purpose of this system is to enable users to manage their household finances hassle-free. The system includes a photographing means, a receiving means, a text information extraction means, an analysis means, a database registration means, a display means, and a push notification means. This allows users to easily grasp their spending situation, obtain future spending forecasts, and obtain advantageous information.

[0040] Explanation of program processing

[0041] Photograph and send receipt

[0042] User

[0043] The user launches the household account book mini-app and takes a picture of the receipt using the camera function, which then sends the image to the server.

[0044] Receipt image reception and analysis

[0045] server

[0046] The server receives the received receipt image. Next, it uses OCR technology to extract text information from the receipt image. The extracted text information is then analyzed and converted into detailed information such as the store name, product name, price, and purchase date and time.

[0047] Classification and storage of generated data

[0048] server

[0049] The server categorizes the analyzed information and stores it in a household accounting database, which is continuously updated to accumulate the user's purchasing history.

[0050] Update and view household accounts

[0051] Terminal

[0052] The terminal receives the latest household accounting information from the server and displays it to the user, allowing the user to easily manage their household finances.

[0053] Accumulation and analysis of purchase history

[0054] server

[0055] The server analyzes the purchase history stored in the household accounting database and generates data such as the purchase frequency of specific stores and products.

[0056] Providing discount information

[0057] server

[0058] The server compares the household accounting database with the advertising database and delivers relevant information via push notifications, which users can use to save money efficiently.

[0059] Specific examples

[0060] For example, if a user goes shopping at a supermarket and receives a receipt, the following happens: The user launches the mini-app and takes a photo of the receipt. The image is sent to the server, and OCR technology extracts data such as "Supermarket X, Tomatoes, 200 yen, October 5, 2023." The analysis means categorizes this data into categories (e.g., "Groceries") and stores it in a database. The user's device displays the data the next day and is also notified that Supermarket X's tomatoes will be on sale over the weekend.

[0061] The above is an embodiment of the present invention, which allows the user to manage their household finances effectively and efficiently.

[0062] The processing flow will be explained below.

[0063] Step 1:

[0064] User

[0065] The user launches the household account book mini-app and selects the camera function. They take a picture of the receipt with the camera and press the capture button. A confirmation screen is then displayed, and the user confirms the image and presses the send button.

[0066] Step 2:

[0067] Terminal

[0068] The device then sends the captured receipt image to the server, where it is encrypted to ensure data security.

[0069] Step 3:

[0070] server

[0071] The server receives the receipt image sent by the user using a receiving means. Once receipt is confirmed, the image is converted into text information using OCR (Optical Character Recognition) technology.

[0072] Step 4:

[0073] server

[0074] The character information extracted by OCR technology is processed by an analysis means to generate detailed information such as the store name, product name, price, purchase date, etc. At this stage, recognition errors are also checked and corrected.

[0075] Step 5:

[0076] server

[0077] The analyzed information is classified by category. For example, it is automatically sorted into categories such as groceries, daily necessities, and beverages. The sorted data is saved in a household accounting database.

[0078] Step 6:

[0079] Terminal

[0080] The device periodically retrieves updated household accounting information from the server, and any new data is automatically reflected in the user's app.

[0081] Step 7:

[0082] User

[0083] Users can check their latest household accounting information within the app and understand their income and expenditure situation. Expenses by category and overall spending status are displayed in graph and list format.

[0084] Step 8:

[0085] server

[0086] The system analyzes a user's purchasing history based on data from the previous month and past data, generating analysis results such as purchase frequency of specific stores and products, and monthly spending trends.

[0087] Step 9:

[0088] server

[0089] The system compares the household accounting database with the advertising database to identify deals relevant to the user. For example, if a frequently purchased item is on sale, that information is extracted.

[0090] Step 10:

[0091] Terminal

[0092] The device receives push notifications from the server and displays them to the user, who can view them within the app and can find specific coupons or sales information.

[0093] Step 11:

[0094] User

[0095] Users can check the notifications and take advantage of the displayed deals, making it easy to save money by using coupons and plan their shopping based on the sale information.

[0096] Example 1

[0097] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0098] Traditional household management methods are time-consuming and require users to manually input data, resulting in problems with accuracy and efficiency. Furthermore, few systems offer added value, such as predicting future spending based on purchase history or providing information on special offers. This makes it difficult for users to effectively manage their household finances and budgets.

[0099] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0100] In this invention, the server includes an image capture means for a user to take a picture of a receipt, an image receiving means for receiving the receipt image captured by the image capture means, a character information extraction means for extracting character information from the receipt image using optical character recognition technology, an information analysis means for analyzing the character information extracted by the character information extraction means and generating a store name, product name, price, and purchase date and time, a database recording means for categorizing the information generated by the information analysis means and storing it in a financial management database, an information display means for displaying the information stored in the financial management database, and a notification means for comparing the financial management database with an advertising database and providing relevant discount information via push notification. This allows users to efficiently manage their household finances simply by taking pictures of receipts, and also enables them to receive future spending forecasts and special offers based on their purchase history.

[0101] "Image capture means" refers to a device or method that allows a user to take a photo of a receipt.

[0102] "Image receiving means" refers to a device or method for receiving a captured receipt image from the user's terminal.

[0103] Optical character recognition (OCR) is a technology that extracts text information from an image.

[0104] "Text information extraction means" refers to a device or method for extracting text information from a receipt image.

[0105] The "information analysis means" refers to a device or method that analyzes the extracted character information and generates detailed information such as the store name, product name, price, and purchase date and time.

[0106] The "database recording means" refers to a device or method for categorizing the analyzed information and storing it in a financial management database.

[0107] "Information display means" refers to a device or method for displaying information stored in the financial management database to a user.

[0108] The "notification means" refers to a device or method that cross-references the financial management database with the advertising database and provides relevant discount information to the user via push notification.

[0109] The present invention is a system that automatically generates a household account book by allowing a user to simply take a photo of a receipt. The following is a specific example for implementing the present invention.

[0110] System Overview

[0111] The purpose of this system is to allow users to manage their household finances hassle-free. The system includes the following components:

[0112] 1. Image acquisition method

[0113] 2. Image Receiving Method

[0114] 3. Optical character recognition technology (OCR)

[0115] 4. Text information extraction method

[0116] 5. Information analysis means

[0117] 6. Database Recording Method

[0118] 7. Information display means

[0119] 8. Means of notification

[0120] Photograph and send receipt

[0121] User

[0122] The user launches the dedicated household accounting mini-app. Next, they take a photo of the receipt using the smartphone's camera. This involves the user operating the camera and adjusting it so that the entire receipt is captured clearly. Once the photo is taken, the app provides instructions for sending the image to the server. The user presses the send button to send the image to the server.

[0123] Receipt image reception and analysis

[0124] server

[0125] The server receives the receipt image sent from the user's device. This reception process includes a method to ensure security using the HTTPS protocol. Next, the server uses the Tesseract OCR library to extract text information from the received image. This OCR process generates text data, which provides detailed information such as the store name, product name, price, and purchase date and time. Furthermore, the extracted text information is structured and organized by category using information analysis means.

[0126] Classification and storage of generated data

[0127] server

[0128] The server classifies the data generated by the information analysis means into categories, such as food, beverages, and daily necessities. This classified data is stored in a financial management database using a database recording means. This database uses a relational database management system such as MySQL or PostgreSQL. The data is continuously updated, and user purchase histories are accumulated.

[0129] Update and view household accounts

[0130] Terminal

[0131] The device periodically accesses the server to obtain the latest household accounting data. Data is obtained via communication via a RESTful API. The received data is temporarily stored in the device's local storage and displayed on a dedicated UI (user interface). This display process uses UI libraries such as React Native. On this screen, the user can view a list of expenses by day and category.

[0132] Accumulation and analysis of purchase history

[0133] server

[0134] The server periodically analyzes the purchase history data stored in the database. This analysis is performed using Python data processing libraries (Pandas and NumPy). For example, graphs are created to show the purchase frequency of specific products and trends in total spending. The results of this analysis are used to generate individual reports for each user.

[0135] Providing discount information

[0136] server

[0137] The server compares the financial management database with the advertising database to generate relevant discount information. This process uses ad distribution services such as the AdSense API. The generated discount information is then provided to users via push notifications. Firebase Cloud Messaging (FCM) is used for push notifications. For example, the server can notify users in real time of sale information for products they frequently purchase.

[0138] Specific examples

[0139] For example, consider the case where a user goes shopping at a supermarket and receives a receipt. The user launches the household accounting mini-app and takes a picture of the receipt using the smartphone's camera. The image is sent to the server, and OCR technology extracts text information such as "Supermarket X, Tomatoes, 200 yen, October 5, 2023." Analysis means categorizes this data into the "grocery" category and stores it in a database. The next day, the user's device receives the latest household accounting data and displays it on the screen. Additionally, a push notification is sent informing them that Supermarket X tomatoes will be on sale over the weekend.

[0140] Example prompts to input to the generative AI model

[0141] Using prompts like the following can effectively leverage generative AI models (e.g., GPT-4):

[0142] "Please explain the automatic household accounting system. The user takes a photo of a receipt and sends the image to a server. The server uses OCR technology to extract text information from the receipt, analyzes it, and saves it in a database. The device receives and displays the latest household accounting data, and the server analyzes purchase history and provides savings information."

[0143] The above is a specific embodiment for carrying out the present invention. This system allows the user to manage their household finances effectively and efficiently.

[0144] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0145] Step 1:

[0146] User

[0147] The user launches the household account book mini-app and takes a photo of the receipt using the smartphone's camera. The user adjusts the camera so that the entire receipt is clearly visible, and then presses the "take a photo" button.

[0148] Input: Receipt image taken by the camera via user operation

[0149] Output: Photographed receipt image data

[0150] Step 2:

[0151] User

[0152] The user presses the "Send" button in the app to send the captured image of the receipt to the server. The app encodes the image data and sends it to the server using the HTTPS protocol.

[0153] Input: Photographed receipt image data

[0154] Output: Receipt image data sent to the server

[0155] Step 3:

[0156] server

[0157] The server receives the sent receipt image. This reception uses the HTTPS protocol to ensure security. The received image data is temporarily stored in server storage.

[0158] Input: Receipt image data sent by the user

[0159] Output: Receipt image data stored on the server

[0160] Step 4:

[0161] server

[0162] The server calls the Tesseract OCR library to extract text information from the received receipt image. The OCR process generates the receipt's text data.

[0163] Input: Receipt image data stored on the server

[0164] Output: Extracted text information (store name, product name, price, purchase date and time, etc.)

[0165] Step 5:

[0166] server

[0167] The server analyzes the text information to identify the store name, product name, price, purchase date, etc. It also categorizes the information into categories, such as food, beverages, and daily necessities.

[0168] Input: Extracted text information

[0169] Output: Parsed and categorized information

[0170] Step 6:

[0171] server

[0172] The server stores the classified information in a financial management database, which also checks for duplicate data. The database uses a relational database management system (MySQL or PostgreSQL).

[0173] Input: Parsed and classified information

[0174] Output: Purchase history information stored in the database

[0175] Step 7:

[0176] Terminal

[0177] The device periodically accesses the server to retrieve the latest household accounting data. This process uses communication via a RESTful API. The retrieved data is temporarily stored in local storage and displayed in a dedicated UI.

[0178] Input: Latest household accounting data obtained from the server

[0179] Output: Household accounting information displayed on the terminal

[0180] Step 8:

[0181] server

[0182] The server periodically analyzes the purchase history data stored in the financial management database. This analysis uses Python data processing libraries (Pandas and NumPy) to generate data such as the purchase frequency of specific products and trends in total spending.

[0183] Input: Purchase history data stored in the financial management database

[0184] Output: Analysis results (purchase frequency, trends in total expenditure, etc.)

[0185] Step 9:

[0186] server

[0187] The server compares the financial management database with the advertising database to generate relevant discount information. This is done using an ad distribution service such as the AdSense API. The generated discount information is then sent to users via push notifications.

[0188] Input: Analysis results, advertising database information

[0189] Output: A push notification with the discount information sent to the user.

[0190] This allows users to efficiently manage their household finances simply by taking photos of receipts, and they can also receive future spending forecasts and special offers based on their purchasing history.

[0191] (Application example 1)

[0192] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0193] In recent years, with the spread of e-commerce, an increasing number of users use multiple online shopping sites. However, manually managing the electronic receipts received from each online shopping site is cumbersome, resulting in a decrease in the efficiency of users' household finance management. In addition, it is difficult to integrate and manage online and offline purchase data, making it difficult to efficiently forecast expenses and manage budgets. This increases the burden on users when managing their household finances.

[0194] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0195] In this invention, the server includes a photographing device for a user to photograph a receipt, a receiving device for receiving the receipt image photographed by the photographing device, a character information extraction device for extracting character information from the receipt image using OCR technology, an analysis device for analyzing the character information extracted by the character information extraction device and generating a store name, product name, price, and purchase date and time, a database registration device for categorizing the information generated by the analysis device and storing it in a household accounting database, a display device for displaying the information stored in the household accounting database, a push notification device for comparing the household accounting database with an advertising database and providing relevant discount information via push notification, and a data acquisition device for automatically acquiring and analyzing electronic receipts from online stores. This allows users to automatically acquire, analyze, and manage not only physical receipts but also electronic receipt information from online stores. This not only enables more efficient household management but also improves users' overall spending forecasting and budget management.

[0196] "Photographing means" refers to the device or function that a user uses to photograph a physical receipt.

[0197] The "receiving means" refers to a device or function for transmitting the receipt image captured by the capturing means to the server and receiving it.

[0198] "Text information extraction means" refers to functions or software for extracting text information from receipt images using OCR technology.

[0199] "Analysis means" refers to a function or software that analyzes the extracted character information and generates detailed information such as the store name, product name, price, and purchase date and time.

[0200] The "database registration means" refers to a function or software that classifies the information generated by the analysis means into categories and stores it in the household accounting database.

[0201] "Display means" refers to devices or software that visually present the information stored in the household accounting database to the user.

[0202] The "push notification means" is a function that compares the household accounting database with the advertisement database and provides relevant discount information to the user as a push notification.

[0203] "Data acquisition means" refers to functions or software for automatically acquiring and analyzing electronic receipts from online stores.

[0204] The present invention provides a system that automatically generates a household account book by allowing users to simply take a photo of a receipt. In particular, the present invention has the ability to automatically acquire and analyze electronic receipts from online stores. The following are specific embodiments for carrying out the present invention.

[0205] System Overview

[0206] The purpose of this system is to enable users to manage their household finances hassle-free. The system includes a photographing means, a receiving means, a text information extraction means, an analysis means, a database registration means, a display means, a push notification means, and a data acquisition means.

[0207] Program processing explanation

[0208] Photograph and send receipt

[0209] The user launches the household accounting app and takes a photo of the physical receipt using the smartphone's camera, which then sends the image to the server.

[0210] Receipt image reception and analysis

[0211] The server receives the receipt image and extracts the text information using an OCR library (Tesseract OCR). The extracted text information is then analyzed and converted into detailed information such as the store name, product name, price, and purchase date and time.

[0212] Classification and storage of generated data

[0213] The server categorizes the analyzed information and stores it in a SQLite database, which is continually updated to accumulate the user's purchasing history.

[0214] Update and view household accounts

[0215] The device receives the latest household accounting information from the server and displays it to the user using React Native, allowing users to easily manage their household finances.

[0216] Automatic capture and analysis of electronic receipts

[0217] The server accesses the user's shopping site account and automatically retrieves the electronic receipt. This process can be set to occur periodically. The retrieved electronic receipt is analyzed using OCR technology to extract text information similar to that of a physical receipt.

[0218] Accumulation and analysis of purchase history

[0219] The server analyzes the purchase history stored in the household account book database and generates data such as the frequency of purchases of specific stores and products, which allows the server to understand users' purchasing patterns and trends.

[0220] Providing discount information

[0221] The server compares the household accounting database with the advertising database and delivers relevant deals via push notifications, such as notifications about sales on specific products.

[0222] Specific examples

[0223] For example, if a user purchases an item from an online store and receives an electronic receipt, the following happens: The server automatically checks the purchase history of the online shopping site periodically and retrieves the electronic receipt, even if the user does not launch the household accounting app. OCR technology extracts information such as "online store, product name, 5000 yen, October 5, 2023," and an analysis method classifies this information into a category (for example, "entertainment") and stores it in a database. When the user opens the app, they can check the latest household accounting information and receive notifications when specific products go on sale.

[0224] Prompt Sentence Examples

[0225] "We will develop an application that analyzes digital receipts received from online stores and reflects them in a household ledger. The process involves obtaining purchase history, extracting text information using OCR technology, storing it in a database, and displaying it in a user interface. The tools used are React Native, Tesseract OCR, and SQLite."

[0226] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0227] Step 1:

[0228] The user launches the household accounting app on their smartphone and takes a photo of the physical receipt. By doing this, the user captures the receipt image using the camera function.

[0229] Input: Physical receipt

[0230] Output: Receipt image

[0231] Step 2:

[0232] The server receives the receipt image sent from the smartphone, and the receipt image is stored on the server as a data packet.

[0233] Input: Receipt image

[0234] Output: Receipt image stored on the server

[0235] Step 3:

[0236] The server uses an OCR library (Tesseract OCR) to extract text information from the received receipt image. During this process, the image data is converted into text data.

[0237] Input: Receipt image

[0238] Output: Extracted text information (store name, product name, price, purchase date and time)

[0239] Step 4:

[0240] The server uses an analysis tool to analyze the extracted text information and generate detailed information such as the store name, product name, price, and purchase date and time, allowing each item to be identified and organized as usable data.

[0241] Input: Extracted text information

[0242] Output: Organized detailed information (store name, product name, price, purchase date and time)

[0243] Step 5:

[0244] The server categorizes the details into categories (e.g., groceries, entertainment) and stores them in an SQLite database. During this process, the data is classified and registered for storage.

[0245] Input: Organized details

[0246] Output: Database records containing information categorized by category

[0247] Step 6:

[0248] The server accesses the user's online shopping site account and automatically retrieves electronic receipts. It periodically checks the purchase history and downloads new electronic receipts.

[0249] Input: User's online shopping site account information

[0250] Output: The obtained e-receipt

[0251] Step 7:

[0252] The server analyzes the acquired electronic receipt using OCR technology to extract text information. Just like physical receipts, text data is generated from electronic receipts.

[0253] Input: Retrieved e-receipt

[0254] Output: Extracted text information (store name, product name, price, purchase date and time)

[0255] Step 8:

[0256] The server categorizes the text information extracted from the electronic receipts and stores it in a database, where it is managed together with the physical receipt data.

[0257] Input: Extracted text information (electronic receipt)

[0258] Output: Database records containing categorized e-receipt information

[0259] Step 9:

[0260] The device receives the latest household accounting data from the server and displays it to the user. React Native is used to provide an intuitive visual interface.

[0261] Input: Latest household accounting data received from the server

[0262] Output: User interface reflecting household accounting data

[0263] Step 10:

[0264] The server compares the household accounting database with the advertising database and pushes relevant deals to users, providing them with useful information in real time.

[0265] Input: household accounting database, advertising database

[0266] Output: Push notification of deals

[0267] By following these steps, users can automatically manage physical receipts and electronic receipts from online stores, and efficiently update and use their household accounts.

[0268] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0269] MODE FOR CARRYING OUT THE INVENTION

[0270] The present invention is a system that automatically generates a household account book when a user simply takes a photo of a receipt, and then uses an emotion engine to recognize the user's emotions, manages household finances, and provides information on special offers. The following is a specific example of how the present invention can be implemented.

[0271] System Overview

[0272] The system includes a camera, a receiver, a text information extraction unit, an analysis unit, a database registration unit, a display unit, a push notification unit, and an emotion engine. The system is designed to allow users to easily manage their household finances and efficiently manage and save money. Furthermore, the system recognizes the user's emotions and adjusts the system's behavior and notification content accordingly, improving ease of use.

[0273] Explanation of program processing

[0274] Photograph and send receipt

[0275] User

[0276] The user launches the household account book mini-app and takes a picture of the receipt using the camera function, which then sends the image to the server.

[0277] Receipt image reception and analysis

[0278] server

[0279] The server receives the received receipt image. Next, it uses OCR technology to extract text information from the receipt image. The extracted text information is then analyzed and converted into detailed information such as the store name, product name, price, and purchase date and time.

[0280] Classification and storage of generated data

[0281] server

[0282] The server categorizes the analyzed information. For example, it automatically sorts items into categories such as groceries, daily necessities, and beverages. The sorted data is then saved in a household accounting database.

[0283] Update and view household accounts

[0284] Terminal

[0285] The device periodically retrieves updated household accounting information from the server, and any new data is automatically reflected in the user's app.

[0286] Accumulation and analysis of purchase history

[0287] server

[0288] The server analyzes the purchase history stored in the household accounting database and generates data such as the purchase frequency of specific stores and products.

[0289] Providing discount information

[0290] server

[0291] The server compares the household accounting database with the advertising database and provides relevant discount information to users via push notifications, which can be used to help users save money efficiently.

[0292] Emotion Engine Operation

[0293] server

[0294] The server runs an emotion engine based on the user's facial and voice data to recognize the user's emotions. For example, it uses a camera and microphone to analyze facial expressions and tone of voice to identify emotions such as joy, surprise, sadness, and anger.

[0295] Adjustments to household account display and notifications

[0296] server

[0297] The server adjusts the visualization of the household accounting data and the content of push notifications based on the user's emotions recognized by the emotion engine. For example, if the user is feeling stressed, it will notify them with an encouraging message or a simple task.

[0298] Specific examples

[0299] For example, when a user goes shopping at a supermarket and receives a receipt, the following sequence of events takes place: The user launches the mini-app and takes a photo of the receipt. The image is sent to the server, where OCR technology extracts data such as "Supermarket X, Tomatoes, 200 yen, October 5, 2023." The analysis tool categorizes this data into categories (e.g., "Groceries") and stores it in a database. The user can check the latest household accounting information displayed on the device and understand spending by category and overall spending. Additionally, if the emotion engine detects stress in the user, an interface is displayed that allows them to easily check their spending for the day, and relevant coupons for the next day are sent via push notification.

[0300] The above is an embodiment of the present invention, which allows users to effectively and efficiently manage their finances and receive support tailored to their emotional state.

[0301] The processing flow will be explained below.

[0302] Step 1:

[0303] User

[0304] The user launches the household account book mini-app and selects the camera function. They take a picture of the receipt with the camera and press the capture button. A confirmation screen is then displayed, and the user confirms the image and presses the send button.

[0305] Step 2:

[0306] Terminal

[0307] The device then sends the captured receipt image to the server, where it is encrypted to ensure data security.

[0308] Step 3:

[0309] server

[0310] The server receives the receipt image sent by the user using a receiving means. Once receipt is confirmed, the image is converted into text information using OCR (Optical Character Recognition) technology.

[0311] Step 4:

[0312] server

[0313] The character information extracted by OCR technology is processed by an analysis means to generate detailed information such as the store name, product name, price, purchase date, etc. At this stage, recognition errors are also checked and corrected.

[0314] Step 5:

[0315] server

[0316] The analyzed information is classified by category. For example, it is automatically sorted into categories such as groceries, daily necessities, and beverages. The sorted data is saved in a household accounting database.

[0317] Step 6:

[0318] Terminal

[0319] The device periodically retrieves updated household accounting information from the server, and any new data is automatically reflected in the user's app.

[0320] Step 7:

[0321] User

[0322] Users can check their latest household accounting information within the app and understand their income and expenditure situation. Expenses by category and overall spending status are displayed in graph and list format.

[0323] Step 8:

[0324] server

[0325] The server runs an emotion engine based on the user's facial and voice data to recognize the user's emotions. For example, it uses a camera and microphone to analyze facial expressions and tone of voice to identify emotions such as joy, surprise, sadness, and anger.

[0326] Step 9:

[0327] server

[0328] The emotion engine adjusts the visualization of household accounting data based on the user's emotions. If the user is feeling stressed, the data display will be simplified and the colors will be softened.

[0329] Step 10:

[0330] server

[0331] The system compares the household accounting database with the advertising database to identify deals relevant to the user. For example, if a frequently purchased item is on sale, that information is extracted.

[0332] Step 11:

[0333] Terminal

[0334] The device receives push notifications from the server and displays them to the user, who can view them within the app and can find specific coupons or sales information.

[0335] Step 12:

[0336] User

[0337] Users can check the notifications and take advantage of the displayed deals, making it easy to save money by using coupons and plan their shopping based on the sale information.

[0338] Step 13:

[0339] server

[0340] The emotion engine periodically analyzes the user's emotions and adjusts the content and timing of push notifications. For example, if the user is feeling stressed, it will provide encouraging messages or information to help them relax.

[0341] The above is a specific embodiment of the present invention that combines an emotion engine, allowing users to effectively manage their finances and receive support according to their emotional state.

[0342] Example 2

[0343] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0344] Modern society demands tools that allow individuals to efficiently and effectively manage their household finances. It is particularly important to easily record and classify daily shopping expenses and use them to predict future spending. Furthermore, providing support that takes into account the user's emotional state would improve user satisfaction and ease of use. However, current household finance management systems rarely meet all of these requirements, and they often place a heavy burden on users. Therefore, the challenge is to provide a system that makes it easy for users to manage their spending and provides the necessary information while taking their emotions into account.

[0345] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0346] In this invention, the server includes: a photographing means for a user to photograph a receipt; a receiving means for receiving the receipt image photographed by the photographing means; a character information extraction means for extracting character information from the receipt image using OCR technology; an analysis means for analyzing the character information extracted by the character information extraction means and generating a store name, product name, price, and purchase date and time; a database registration means for categorizing the information generated by the analysis means and storing it in a database; a display means for displaying the information stored in the database; a push notification means for comparing the database with other information databases and providing highly relevant information via push notification; an emotion recognition means for analyzing the user's facial expressions and voice data to recognize emotions; and an adjustment means for adjusting the display content of the display means and the notification content of the push notification means based on the emotion recognized by the emotion recognition means. This not only allows users to easily record and manage their expenses, but also provides them with optimal information according to their emotional state, making household management more efficient and effective.

[0347] "Photographing means" refers to a function or device that allows a user to photograph a receipt with a camera.

[0348] The "receiving means" refers to a function or device for sending image data of a photographed receipt to a server and receiving it.

[0349] "Text information extraction means" refers to the functions and techniques for extracting text information from receipt images using OCR technology.

[0350] The "analysis means" refers to a function or technology for analyzing the character information acquired by the character information extraction means and converting it into detailed data such as the store name, product name, price, and purchase date and time.

[0351] The "database registration means" refers to the function or technology for classifying the information generated by the analysis means into categories and storing it in a database.

[0352] "Display means" refers to functions and technologies for visually presenting information stored in a database to a user.

[0353] "Push notification means" refers to a function or technology that compares the household accounting database with other information databases and provides relevant information to the user via push notification.

[0354] "Emotion recognition means" refers to functions and technologies for analyzing a user's facial expressions and voice data and recognizing the user's emotions.

[0355] The "adjustment means" refers to a function or technology for adjusting the display content of the display means or the notification content of the push notification means based on the emotion recognized by the emotion recognition means.

[0356] MODE FOR CARRYING OUT THE INVENTION

[0357] The present invention is a system that automatically generates a household account book simply by a user taking a photo of a receipt, and then uses an emotion engine to recognize the user's emotions, providing household management and related information. Specific means for implementing the present invention and examples of its operation are shown below.

[0358] System Overview

[0359] The system includes a camera, a receiver, a text information extractor, an analyzer, a database registerer, a displayer, a push notification system, and an emotion recognition system. The system is designed to allow users to easily manage their household finances and achieve efficient spending management and savings. Furthermore, the system's usability is enhanced by recognizing the user's emotions and adjusting the system's behavior and notification content accordingly.

[0360] Photograph and send receipt

[0361] User

[0362] The user launches the household accounting mini-app on their smartphone or tablet and takes a picture of the receipt using the device's camera. For example, they can press the "take a picture of receipt" button on the app's home screen to switch to the camera screen. When the user focuses on the receipt and presses the shutter button, the captured image is sent to the server.

[0363] Receipt image reception and analysis

[0364] server

[0365] The server receives the receipt image sent by the user. It then uses OCR technology such as Tesseract to extract text information from the image. For example, a string of characters in the format "Super X, Tomato, ¥200, October 5, 2023" is generated. This text information is then broken down into detailed data for the database using an analysis tool.

[0366] Classification and storage of generated data

[0367] server

[0368] The server classifies the analyzed data into categories such as "grocery" and "daily necessities." For example, data classified as "grocery" is stored in a MySQL database.

[0369] Update and view household accounts

[0370] Terminal

[0371] The device periodically sends a request to the server to retrieve updated information. If new data is available, it is automatically reflected in the on-screen household ledger display, allowing the user to view it in real time.

[0372] Accumulation and analysis of purchase history

[0373] server

[0374] The server collects past purchase data and analyzes it based on specific items (e.g., by store, frequency of product purchases, etc.). The results of this analysis visualize the purchasing trends of each user.

[0375] Providing discount information

[0376] server

[0377] The server compares the household account data with the advertising database to obtain information such as "Products at a specific store are 10% off this week." This information is then sent to the user as a push notification.

[0378] Operation of emotion recognition means

[0379] server

[0380] The server analyzes the user's facial expressions and voice data in real time to recognize emotions. Using a deep learning model, the user's emotional state is classified into categories such as "surprise," "sadness," and "happiness."

[0381] Adjustments to household account display and notifications

[0382] server

[0383] If the server determines that the user is feeling stressed based on the analysis results of the emotion recognition means, it will provide a visually easy-to-read interface and send push notifications with information on benefits for relaxation.

[0384] Specific examples

[0385] For example, when a user goes shopping at a supermarket and receives a receipt, the following process takes place: The user launches the household accounting mini-app and takes a photo of the receipt. The image is sent to the server, and OCR technology extracts the data: "Supermarket X, Tomatoes, 200 yen, October 5, 2023." The analysis means classifies this data into categories (e.g., "Groceries") and stores it in a database. The user can check the latest household accounting information on their device and understand their spending status by category and overall. Furthermore, if the emotion recognition means detects that the user is stressed, an interface is provided that allows them to easily check their spending status for that day, and relevant coupons for the next day are pushed to them.

[0386] Prompt Sentence Examples

[0387] "Please explain the process of a system that analyzes receipt images taken by users and automatically updates the household account book."

[0388] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0389] Step 1:

[0390] Photograph and send receipt

[0391] The user launches the household account book mini-app and clicks the "Take a photo of receipt" button on the home screen, which launches the camera screen.

[0392] The user focuses on the receipt and presses the shutter button to take a photo of the receipt.

[0393] Input: A receipt image captured by the camera.

[0394] After confirming the image, the user presses the "send" button, and the captured image is sent to the server.

[0395] Output: Receipt image data sent to the server.

[0396] Step 2:

[0397] Receipt image reception and analysis

[0398] The server receives the receipt image sent by the user.

[0399] Input: Receipt image data sent from the user device.

[0400] The server uses OCR technology (e.g., Tesseract OCR) to extract text information from the receipt image.

[0401] Data processing: The process of extracting text information from receipt image data.

[0402] The server analyzes the extracted text information and converts it into detailed data such as the store name, product name, price, and purchase date and time.

[0403] Output: Text information such as store name, product name, price, purchase date and time.

[0404] Step 3:

[0405] Classification and storage of generated data

[0406] The server categorizes the analyzed text information into categories, such as "foodstuffs" and "daily necessities."

[0407] Input: Analyzed text information (store name, product name, price, purchase date and time, etc.).

[0408] The server stores this information in a database (e.g., a MySQL database).

[0409] Data processing: The process of classifying data by category and storing it in a database.

[0410] Output: Database storage of character information categorized by category.

[0411] Step 4:

[0412] Update and view household accounts

[0413] The device periodically sends requests to the server to obtain updated household accounting information.

[0414] Input: Updated household accounting information provided by the server.

[0415] If the device has new data, it will automatically be reflected in the household ledger display on the screen.

[0416] Data processing: Display processing of new household accounting data.

[0417] Output: The updated household accounting information is displayed on the user's device.

[0418] Step 5:

[0419] Accumulation and analysis of purchase history

[0420] The server collects the purchase history stored in the household account book database and performs analysis based on specific items (e.g., by store, frequency of purchase of product, etc.).

[0421] Input: Past purchase history stored in the household accounting database.

[0422] The server aggregates this data and visualizes each user's purchasing trends.

[0423] Data processing: Analysis and aggregation of purchase history data.

[0424] Output: Purchasing trend data based on the analysis results.

[0425] Step 6:

[0426] Providing discount information

[0427] The server compares the household account book database with other information databases to obtain discount information relevant to the user.

[0428] Input: Information from the household accounting database and advertising database.

[0429] The server provides relevant information to the user via push notifications.

[0430] Data processing: Cross-database information matching and notification content generation.

[0431] Output: Deals sent via push notification.

[0432] Step 7:

[0433] Operation of emotion recognition means

[0434] The user allows access to the device's camera and microphone.

[0435] The server uses emotion recognition means to analyze the user's facial expressions and voice data and recognize emotions.

[0436] Input: User's facial expression data and voice data.

[0437] The server uses a deep learning model to identify the emotional state.

[0438] Data processing: Analysis and processing of facial expression and voice data.

[0439] Output: Identified emotional state data.

[0440] Step 8:

[0441] Adjustments to household account display and notifications

[0442] The server adjusts the visualization method of the household accounting data and the notification content based on the results of the emotion recognition means.

[0443] Input: Identified emotional state data and household ledger information.

[0444] The server generates an interface and notification content that matches the user's emotions.

[0445] Data processing: Adjustment of display and notification content based on emotional state.

[0446] Output: Adjusted interface and push notification content.

[0447] (Application example 2)

[0448] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0449] In conventional household management systems, entering expenditure data is tedious, and it is particularly difficult to fill out receipts while driving or on the go. Furthermore, the systems do not take the user's feelings into consideration, resulting in poor usability. Furthermore, they do not adequately provide advertisements or discount information linked to receipt information, making it difficult for users to save money efficiently. To solve these problems, it is necessary to improve the user experience in household management systems.

[0450] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0451] In this invention, the server includes a photographing means for a user to photograph a receipt, a receiving means for receiving the receipt image photographed by the photographing means, a character information extraction means for extracting character information from the receipt image using OCR technology, an analysis means for analyzing the character information extracted by the character information extraction means and generating a store name, product name, price, and purchase date and time, a database registration means for categorizing the information generated by the analysis means and storing it in a household accounting database, a display means for displaying the information stored in the household accounting database, a push notification means for comparing the household accounting database with an advertising database and providing relevant discount information via push notification, an emotion recognition means for recognizing the user's emotion, an adjustment means for adjusting the display method of the household accounting data and the notification content based on the emotion analyzed by the emotion recognition means, and an integration means integrated into the infotainment system of an autonomous vehicle and coordinating with an in-vehicle camera and a voice assistant. This enables safe household management while driving and provides optimal information tailored to the user's emotion.

[0452] A "photographing means" is a device used by a user to capture an image of a printed medium such as a receipt.

[0453] The "receiving means" is a function for importing the receipt image captured by the image capturing means into the system.

[0454] The "character information extraction means" is a function that uses OCR technology to recognize and extract characters from a captured receipt image.

[0455] The "analysis means" is a function that analyzes the extracted character information and converts it into specific data such as the store name, product name, price, and purchase date and time.

[0456] The "database registration means" is a function for classifying the information generated by the analysis means into categories and storing the information in the household account book database.

[0457] The "display means" is a device or function for visually presenting the information stored in the household accounting database to the user.

[0458] The "push notification means" is a function that checks the household account book database against the advertisement database and notifies the user of relevant deals.

[0459] The "emotion recognition means" is a function for analyzing and recognizing emotions from the user's voice, facial expressions, etc.

[0460] The "adjustment means" is a function that adjusts the display method of household accounting data and notification content based on the user's emotions analyzed by the emotion recognition means.

[0461] "Integration means" refers to the functionality for integrating and linking other systems and devices with the infotainment system of an autonomous vehicle.

[0462] The present invention is a system that is integrated into the infotainment system of an autonomous vehicle, allowing users to safely and efficiently manage their household finances while driving. The system mainly includes a photographing means, a receiving means, a text information extraction means, an analysis means, a database registration means, a display means, a push notification means, an emotion recognition means, an adjustment means, and an integration means.

[0463] Hardware and Software Used

[0464] Hardware:

[0465] In-car camera

[0466] microphone

[0467] Touch panel display

[0468] In-vehicle computer

[0469] software:

[0470] OCR engine (e.g. Tesseract)

[0471] Emotion recognition engine (e.g. Microsoft Azure Cognitive Services)

[0472] Speech synthesis engine (e.g. Google Text-to-Speech API)

[0473] Cloud databases (e.g. Firebase)

[0474] Infotainment System API

[0475] System Operation

[0476] User

[0477] After shopping, users can take a photo of the receipt with the car's camera, ensuring safety while driving.

[0478] server

[0479] 1. Receipt reception and OCR processing:

[0480] The receiving device captures the receipt image taken by the in-car camera, and the OCR engine extracts text information from the image. This process generates data such as the store name, product name, price, and purchase date and time.

[0481] 2. Data analysis and classification:

[0482] The analysis means analyzes various data using the extracted character information, and the database registration means stores the information in the household account book database. At this time, the data is automatically classified into categories such as food, daily necessities, and beverages.

[0483] 3. Matching purchase history with advertising data:

[0484] The server compares the household account database with the advertisement database to generate relevant deals for the user, which are then provided to the user via push notification.

[0485] 4. Emotion recognition:

[0486] An emotion recognition engine analyzes voice data collected through microphones in the car, thereby identifying the user's current emotion (happiness, surprise, sadness, anger, etc.).

[0487] 5. Notification Adjustments:

[0488] The application adjusts the display method of household accounting data and notification content based on the emotion information acquired by the emotion recognition means, thereby providing optimal information suited to the user.

[0489] Terminal

[0490] The device (touch panel display of the infotainment system) periodically retrieves the latest household accounting information sent from the server and visually displays it. It also has a function to read out the contents of push notifications using a speech synthesis engine.

[0491] Specific examples

[0492] For example, if a user is driving an autonomous vehicle and purchases coffee at a drive-thru and receives a receipt, they simply present the receipt to the in-car camera. The onboard computer uses an OCR engine to extract information such as "coffee shop, coffee, 400 yen, October 10, 2023." The data is then stored and categorized in a Firebase cloud database. The infotainment system's touchscreen display displays the latest household finances and provides relevant deals via voice. If the emotion recognition engine determines the user is stressed, the system will notify them with a gentle voice message encouraging them to relax.

[0493] Prompt Sentence Examples

[0494] When a user takes a photo of a receipt with the in-car camera, the OCR engine extracts the text information and stores it in Firebase. The emotion recognition engine analyzes the user's voice and identifies their emotion. Next, the household accounting information is displayed on the touch panel, and if necessary, the speech synthesis engine provides discount information.

[0495] This system allows users to intuitively and safely manage their household finances while driving, and provides optimal information based on their emotional state. This technology is specialized for autonomous vehicles, contributing to an improved user experience.

[0496] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0497] Step 1:

[0498] The user takes a photo of the receipt

[0499] After shopping, the user presents the receipt to the camera in the autonomous vehicle and takes a photo. The input is the receipt image, and the output is the transfer of image data to the on-board computer. User interaction in this step is kept to a minimum.

[0500] Step 2:

[0501] Receiving receipt images

[0502] The server (on-board computer) receives receipt images taken by the user with the in-car camera. The input is the receipt image, and the output is image data provided to the OCR engine. This step ensures that the receipt image is captured in the system.

[0503] Step 3:

[0504] OCR analysis

[0505] Extract text information from the receipt image using an OCR engine (e.g., Tesseract). The input is the receipt image, and the output is the extracted text information (store name, product name, price, purchase date and time). This step converts the image data into text data.

[0506] Step 4:

[0507] Data analysis

[0508] The server analyzes the extracted text and generates the appropriate information (store name, product name, price, purchase date and time). The input is text, and the output is the analyzed data. This step involves formatting and categorizing the data.

[0509] Step 5:

[0510] Saving to a database

[0511] The database registration means classifies the analyzed information by category and stores it in the household account book database. The input is the analyzed data, and the output is the database registration of the classified data. This allows the user to check their expenses by category.

[0512] Step 6:

[0513] Displaying household accounts

[0514] The terminal (the display of the in-vehicle infotainment system) periodically retrieves the latest household accounting information from the server and displays it. The input is information from the household accounting database, and the output is a visual presentation of the information on the display. At this step, the user can check the expenditure information.

[0515] Step 7:

[0516] Matching purchase history with advertising data

[0517] The server compares the household account database with the advertisement database and generates deals related to the user. The input is household account data and advertisement data, and the output is deals. This step generates information useful to the user.

[0518] Step 8:

[0519] Push notifications

[0520] The push notification means notifies the user of the discount information generated in the previous step. The input is the discount information, and the output is a real-time notification to the user. In this step, the user receives the information efficiently.

[0521] Step 9:

[0522] emotion recognition

[0523] The emotion recognition means analyzes the voice data collected using the in-car microphone and identifies the user's emotion. The input is voice data, and the output is the user's emotional information. This step identifies the user's emotional state.

[0524] Step 10:

[0525] Adjusting notification content

[0526] The server adjusts the display method of the household accounting data and the notification content based on the emotional information analyzed by the emotion recognition means. The input is the user's emotional information, and the output is the adjusted display method and notification content. This step provides the user with the most appropriate information.

[0527] The above is a detailed description of each processing step, which allows the system to provide advanced household management and emotion-based services while ensuring user safety.

[0528] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0529] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0530] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0531] [Second embodiment]

[0532] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0533] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0534] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0535] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0536] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0537] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0538] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0539] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0540] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0541] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0542] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0543] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0544] MODE FOR CARRYING OUT THE INVENTION

[0545] The present invention provides a system that automatically generates a household account book by allowing a user to simply take a photo of a receipt. The following is a specific example for implementing the present invention.

[0546] System Overview

[0547] The purpose of this system is to enable users to manage their household finances hassle-free. The system includes a photographing means, a receiving means, a text information extraction means, an analysis means, a database registration means, a display means, and a push notification means. This allows users to easily grasp their spending situation, obtain future spending forecasts, and obtain advantageous information.

[0548] Explanation of program processing

[0549] Photograph and send receipt

[0550] User

[0551] The user launches the household account book mini-app and takes a picture of the receipt using the camera function, which then sends the image to the server.

[0552] Receipt image reception and analysis

[0553] server

[0554] The server receives the received receipt image. Next, it uses OCR technology to extract text information from the receipt image. The extracted text information is then analyzed and converted into detailed information such as the store name, product name, price, and purchase date and time.

[0555] Classification and storage of generated data

[0556] server

[0557] The server categorizes the analyzed information and stores it in a household accounting database, which is continuously updated to accumulate the user's purchasing history.

[0558] Update and view household accounts

[0559] Terminal

[0560] The terminal receives the latest household accounting information from the server and displays it to the user, allowing the user to easily manage their household finances.

[0561] Accumulation and analysis of purchase history

[0562] server

[0563] The server analyzes the purchase history stored in the household accounting database and generates data such as the purchase frequency of specific stores and products.

[0564] Providing discount information

[0565] server

[0566] The server compares the household accounting database with the advertising database and delivers relevant information via push notifications, which users can use to save money efficiently.

[0567] Specific examples

[0568] For example, if a user goes shopping at a supermarket and receives a receipt, the following happens: The user launches the mini-app and takes a photo of the receipt. The image is sent to the server, and OCR technology extracts data such as "Supermarket X, Tomatoes, 200 yen, October 5, 2023." The analysis means categorizes this data into categories (e.g., "Groceries") and stores it in a database. The user's device displays the data the next day and is also notified that Supermarket X's tomatoes will be on sale over the weekend.

[0569] The above is an embodiment of the present invention, which allows the user to manage their household finances effectively and efficiently.

[0570] The processing flow will be explained below.

[0571] Step 1:

[0572] User

[0573] The user launches the household account book mini-app and selects the camera function. They take a picture of the receipt with the camera and press the capture button. A confirmation screen is then displayed, and the user confirms the image and presses the send button.

[0574] Step 2:

[0575] Terminal

[0576] The device then sends the captured receipt image to the server, where it is encrypted to ensure data security.

[0577] Step 3:

[0578] server

[0579] The server receives the receipt image sent by the user using a receiving means. Once receipt is confirmed, the image is converted into text information using OCR (Optical Character Recognition) technology.

[0580] Step 4:

[0581] server

[0582] The character information extracted by OCR technology is processed by an analysis means to generate detailed information such as the store name, product name, price, purchase date, etc. At this stage, recognition errors are also checked and corrected.

[0583] Step 5:

[0584] server

[0585] The analyzed information is classified by category. For example, it is automatically sorted into categories such as groceries, daily necessities, and beverages. The sorted data is saved in a household accounting database.

[0586] Step 6:

[0587] Terminal

[0588] The device periodically retrieves updated household accounting information from the server, and any new data is automatically reflected in the user's app.

[0589] Step 7:

[0590] User

[0591] Users can check their latest household accounting information within the app and understand their income and expenditure situation. Expenses by category and overall spending status are displayed in graph and list format.

[0592] Step 8:

[0593] server

[0594] The system analyzes a user's purchasing history based on data from the previous month and past data, generating analysis results such as purchase frequency of specific stores and products, and monthly spending trends.

[0595] Step 9:

[0596] server

[0597] The system compares the household accounting database with the advertising database to identify deals relevant to the user. For example, if a frequently purchased item is on sale, that information is extracted.

[0598] Step 10:

[0599] Terminal

[0600] The device receives push notifications from the server and displays them to the user, who can view them within the app and can find specific coupons or sales information.

[0601] Step 11:

[0602] User

[0603] Users can check the notifications and take advantage of the displayed deals, making it easy to save money by using coupons and plan their shopping based on the sale information.

[0604] Example 1

[0605] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0606] Traditional household management methods are time-consuming and require users to manually input data, resulting in problems with accuracy and efficiency. Furthermore, few systems offer added value, such as predicting future spending based on purchase history or providing information on special offers. This makes it difficult for users to effectively manage their household finances and budgets.

[0607] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0608] In this invention, the server includes an image capture means for a user to take a picture of a receipt, an image receiving means for receiving the receipt image captured by the image capture means, a character information extraction means for extracting character information from the receipt image using optical character recognition technology, an information analysis means for analyzing the character information extracted by the character information extraction means and generating a store name, product name, price, and purchase date and time, a database recording means for categorizing the information generated by the information analysis means and storing it in a financial management database, an information display means for displaying the information stored in the financial management database, and a notification means for comparing the financial management database with an advertising database and providing relevant discount information via push notification. This allows users to efficiently manage their household finances simply by taking pictures of receipts, and also enables them to receive future spending forecasts and special offers based on their purchase history.

[0609] "Image capture means" refers to a device or method that allows a user to take a photo of a receipt.

[0610] "Image receiving means" refers to a device or method for receiving a captured receipt image from the user's terminal.

[0611] Optical character recognition (OCR) is a technology that extracts text information from an image.

[0612] "Text information extraction means" refers to a device or method for extracting text information from a receipt image.

[0613] The "information analysis means" refers to a device or method that analyzes the extracted character information and generates detailed information such as the store name, product name, price, and purchase date and time.

[0614] The "database recording means" refers to a device or method for categorizing the analyzed information and storing it in a financial management database.

[0615] "Information display means" refers to a device or method for displaying information stored in the financial management database to a user.

[0616] The "notification means" refers to a device or method that cross-references the financial management database with the advertising database and provides relevant discount information to the user via push notification.

[0617] The present invention is a system that automatically generates a household account book by allowing a user to simply take a photo of a receipt. The following is a specific example for implementing the present invention.

[0618] System Overview

[0619] The purpose of this system is to allow users to manage their household finances hassle-free. The system includes the following components:

[0620] 1. Image acquisition method

[0621] 2. Image Receiving Method

[0622] 3. Optical character recognition technology (OCR)

[0623] 4. Text information extraction method

[0624] 5. Information analysis means

[0625] 6. Database Recording Method

[0626] 7. Information display means

[0627] 8. Means of notification

[0628] Photograph and send receipt

[0629] User

[0630] The user launches the dedicated household accounting mini-app. Next, they take a photo of the receipt using the smartphone's camera. This involves the user operating the camera and adjusting it so that the entire receipt is captured clearly. Once the photo is taken, the app provides instructions for sending the image to the server. The user presses the send button to send the image to the server.

[0631] Receipt image reception and analysis

[0632] server

[0633] The server receives the receipt image sent from the user's device. This reception process includes a method to ensure security using the HTTPS protocol. Next, the server uses the Tesseract OCR library to extract text information from the received image. This OCR process generates text data, which provides detailed information such as the store name, product name, price, and purchase date and time. Furthermore, the extracted text information is structured and organized by category using information analysis means.

[0634] Classification and storage of generated data

[0635] server

[0636] The server classifies the data generated by the information analysis means into categories, such as food, beverages, and daily necessities. This classified data is stored in a financial management database using a database recording means. This database uses a relational database management system such as MySQL or PostgreSQL. The data is continuously updated, and user purchase histories are accumulated.

[0637] Update and view household accounts

[0638] Terminal

[0639] The device periodically accesses the server to obtain the latest household accounting data. Data is obtained via communication via a RESTful API. The received data is temporarily stored in the device's local storage and displayed on a dedicated UI (user interface). This display process uses UI libraries such as React Native. On this screen, the user can view a list of expenses by day and category.

[0640] Accumulation and analysis of purchase history

[0641] server

[0642] The server periodically analyzes the purchase history data stored in the database. This analysis is performed using Python data processing libraries (Pandas and NumPy). For example, graphs are created to show the purchase frequency of specific products and trends in total spending. The results of this analysis are used to generate individual reports for each user.

[0643] Providing discount information

[0644] server

[0645] The server compares the financial management database with the advertising database to generate relevant discount information. This process uses ad distribution services such as the AdSense API. The generated discount information is then provided to users via push notifications. Firebase Cloud Messaging (FCM) is used for push notifications. For example, the server can notify users in real time of sale information for products they frequently purchase.

[0646] Specific examples

[0647] For example, consider the case where a user goes shopping at a supermarket and receives a receipt. The user launches the household accounting mini-app and takes a picture of the receipt using the smartphone's camera. The image is sent to the server, and OCR technology extracts text information such as "Supermarket X, Tomatoes, 200 yen, October 5, 2023." Analysis means categorizes this data into the "grocery" category and stores it in a database. The next day, the user's device receives the latest household accounting data and displays it on the screen. Additionally, a push notification is sent informing them that Supermarket X tomatoes will be on sale over the weekend.

[0648] Example prompts to input to the generative AI model

[0649] Using prompts like the following can effectively leverage generative AI models (e.g., GPT-4):

[0650] "Please explain the automatic household accounting system. The user takes a photo of a receipt and sends the image to a server. The server uses OCR technology to extract text information from the receipt, analyzes it, and saves it in a database. The device receives and displays the latest household accounting data, and the server analyzes purchase history and provides savings information."

[0651] The above is a specific embodiment for carrying out the present invention. This system allows the user to manage their household finances effectively and efficiently.

[0652] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0653] Step 1:

[0654] User

[0655] The user launches the household account book mini-app and takes a photo of the receipt using the smartphone's camera. The user adjusts the camera so that the entire receipt is clearly visible, and then presses the "take a photo" button.

[0656] Input: Receipt image taken by the camera via user operation

[0657] Output: Photographed receipt image data

[0658] Step 2:

[0659] User

[0660] The user presses the "Send" button in the app to send the captured image of the receipt to the server. The app encodes the image data and sends it to the server using the HTTPS protocol.

[0661] Input: Photographed receipt image data

[0662] Output: Receipt image data sent to the server

[0663] Step 3:

[0664] server

[0665] The server receives the sent receipt image. This reception uses the HTTPS protocol to ensure security. The received image data is temporarily stored in server storage.

[0666] Input: Receipt image data sent by the user

[0667] Output: Receipt image data stored on the server

[0668] Step 4:

[0669] server

[0670] The server calls the Tesseract OCR library to extract text information from the received receipt image. The OCR process generates the receipt's text data.

[0671] Input: Receipt image data stored on the server

[0672] Output: Extracted text information (store name, product name, price, purchase date and time, etc.)

[0673] Step 5:

[0674] server

[0675] The server analyzes the text information to identify the store name, product name, price, purchase date, etc. It also categorizes the information into categories, such as food, beverages, and daily necessities.

[0676] Input: Extracted text information

[0677] Output: Parsed and categorized information

[0678] Step 6:

[0679] server

[0680] The server stores the classified information in a financial management database, which also checks for duplicate data. The database uses a relational database management system (MySQL or PostgreSQL).

[0681] Input: Parsed and classified information

[0682] Output: Purchase history information stored in the database

[0683] Step 7:

[0684] Terminal

[0685] The device periodically accesses the server to retrieve the latest household accounting data. This process uses communication via a RESTful API. The retrieved data is temporarily stored in local storage and displayed in a dedicated UI.

[0686] Input: Latest household accounting data obtained from the server

[0687] Output: Household accounting information displayed on the terminal

[0688] Step 8:

[0689] server

[0690] The server periodically analyzes the purchase history data stored in the financial management database. This analysis uses Python data processing libraries (Pandas and NumPy) to generate data such as the purchase frequency of specific products and trends in total spending.

[0691] Input: Purchase history data stored in the financial management database

[0692] Output: Analysis results (purchase frequency, trends in total expenditure, etc.)

[0693] Step 9:

[0694] server

[0695] The server compares the financial management database with the advertising database to generate relevant discount information. This is done using an ad distribution service such as the AdSense API. The generated discount information is then sent to users via push notifications.

[0696] Input: Analysis results, advertising database information

[0697] Output: A push notification with the discount information sent to the user.

[0698] This allows users to efficiently manage their household finances simply by taking photos of receipts, and they can also receive future spending forecasts and special offers based on their purchasing history.

[0699] (Application example 1)

[0700] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0701] In recent years, with the spread of e-commerce, an increasing number of users use multiple online shopping sites. However, manually managing the electronic receipts received from each online shopping site is cumbersome, resulting in a decrease in the efficiency of users' household finance management. In addition, it is difficult to integrate and manage online and offline purchase data, making it difficult to efficiently forecast expenses and manage budgets. This increases the burden on users when managing their household finances.

[0702] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0703] In this invention, the server includes a photographing device for a user to photograph a receipt, a receiving device for receiving the receipt image photographed by the photographing device, a character information extraction device for extracting character information from the receipt image using OCR technology, an analysis device for analyzing the character information extracted by the character information extraction device and generating a store name, product name, price, and purchase date and time, a database registration device for categorizing the information generated by the analysis device and storing it in a household accounting database, a display device for displaying the information stored in the household accounting database, a push notification device for comparing the household accounting database with an advertising database and providing relevant discount information via push notification, and a data acquisition device for automatically acquiring and analyzing electronic receipts from online stores. This allows users to automatically acquire, analyze, and manage not only physical receipts but also electronic receipt information from online stores. This not only enables more efficient household management but also improves users' overall spending forecasting and budget management.

[0704] "Photographing means" refers to the device or function that a user uses to photograph a physical receipt.

[0705] The "receiving means" refers to a device or function for transmitting the receipt image captured by the capturing means to the server and receiving it.

[0706] "Text information extraction means" refers to functions or software for extracting text information from receipt images using OCR technology.

[0707] "Analysis means" refers to a function or software that analyzes the extracted character information and generates detailed information such as the store name, product name, price, and purchase date and time.

[0708] The "database registration means" refers to a function or software that classifies the information generated by the analysis means into categories and stores it in the household accounting database.

[0709] "Display means" refers to devices or software that visually present the information stored in the household accounting database to the user.

[0710] The "push notification means" is a function that compares the household accounting database with the advertisement database and provides relevant discount information to the user as a push notification.

[0711] "Data acquisition means" refers to functions or software for automatically acquiring and analyzing electronic receipts from online stores.

[0712] The present invention provides a system that automatically generates a household account book by allowing users to simply take a photo of a receipt. In particular, the present invention has the ability to automatically acquire and analyze electronic receipts from online stores. The following are specific embodiments for carrying out the present invention.

[0713] System Overview

[0714] The purpose of this system is to enable users to manage their household finances hassle-free. The system includes a photographing means, a receiving means, a text information extraction means, an analysis means, a database registration means, a display means, a push notification means, and a data acquisition means.

[0715] Program processing explanation

[0716] Photograph and send receipt

[0717] The user launches the household accounting app and takes a photo of the physical receipt using the smartphone's camera, which then sends the image to the server.

[0718] Receipt image reception and analysis

[0719] The server receives the receipt image and extracts the text information using an OCR library (Tesseract OCR). The extracted text information is then analyzed and converted into detailed information such as the store name, product name, price, and purchase date and time.

[0720] Classification and storage of generated data

[0721] The server categorizes the analyzed information and stores it in a SQLite database, which is continually updated to accumulate the user's purchasing history.

[0722] Update and view household accounts

[0723] The device receives the latest household accounting information from the server and displays it to the user using React Native, allowing users to easily manage their household finances.

[0724] Automatic capture and analysis of electronic receipts

[0725] The server accesses the user's shopping site account and automatically retrieves the electronic receipt. This process can be set to occur periodically. The retrieved electronic receipt is analyzed using OCR technology to extract text information similar to that of a physical receipt.

[0726] Accumulation and analysis of purchase history

[0727] The server analyzes the purchase history stored in the household account book database and generates data such as the frequency of purchases of specific stores and products, which allows the server to understand users' purchasing patterns and trends.

[0728] Providing discount information

[0729] The server compares the household accounting database with the advertising database and delivers relevant deals via push notifications, such as notifications about sales on specific products.

[0730] Specific examples

[0731] For example, if a user purchases an item from an online store and receives an electronic receipt, the following happens: The server automatically checks the purchase history of the online shopping site periodically and retrieves the electronic receipt, even if the user does not launch the household accounting app. OCR technology extracts information such as "online store, product name, 5000 yen, October 5, 2023," and an analysis method classifies this information into a category (for example, "entertainment") and stores it in a database. When the user opens the app, they can check the latest household accounting information and receive notifications when specific products go on sale.

[0732] Prompt Sentence Examples

[0733] "We will develop an application that analyzes digital receipts received from online stores and reflects them in a household ledger. The process involves obtaining purchase history, extracting text information using OCR technology, storing it in a database, and displaying it in a user interface. The tools used are React Native, Tesseract OCR, and SQLite."

[0734] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0735] Step 1:

[0736] The user launches the household accounting app on their smartphone and takes a photo of the physical receipt. By doing this, the user captures the receipt image using the camera function.

[0737] Input: Physical receipt

[0738] Output: Receipt image

[0739] Step 2:

[0740] The server receives the receipt image sent from the smartphone, and the receipt image is stored on the server as a data packet.

[0741] Input: Receipt image

[0742] Output: Receipt image stored on the server

[0743] Step 3:

[0744] The server uses an OCR library (Tesseract OCR) to extract text information from the received receipt image. During this process, the image data is converted into text data.

[0745] Input: Receipt image

[0746] Output: Extracted text information (store name, product name, price, purchase date and time)

[0747] Step 4:

[0748] The server uses an analysis tool to analyze the extracted text information and generate detailed information such as the store name, product name, price, and purchase date and time, allowing each item to be identified and organized as usable data.

[0749] Input: Extracted text information

[0750] Output: Organized detailed information (store name, product name, price, purchase date and time)

[0751] Step 5:

[0752] The server categorizes the details into categories (e.g., groceries, entertainment) and stores them in an SQLite database. During this process, the data is classified and registered for storage.

[0753] Input: Organized details

[0754] Output: Database records containing information categorized by category

[0755] Step 6:

[0756] The server accesses the user's online shopping site account and automatically retrieves electronic receipts. It periodically checks the purchase history and downloads new electronic receipts.

[0757] Input: User's online shopping site account information

[0758] Output: The obtained e-receipt

[0759] Step 7:

[0760] The server analyzes the acquired electronic receipt using OCR technology to extract text information. Just like physical receipts, text data is generated from electronic receipts.

[0761] Input: Retrieved e-receipt

[0762] Output: Extracted text information (store name, product name, price, purchase date and time)

[0763] Step 8:

[0764] The server categorizes the text information extracted from the electronic receipts and stores it in a database, where it is managed together with the physical receipt data.

[0765] Input: Extracted text information (electronic receipt)

[0766] Output: Database records containing categorized e-receipt information

[0767] Step 9:

[0768] The device receives the latest household accounting data from the server and displays it to the user. React Native is used to provide an intuitive visual interface.

[0769] Input: Latest household accounting data received from the server

[0770] Output: User interface reflecting household accounting data

[0771] Step 10:

[0772] The server compares the household accounting database with the advertising database and pushes relevant deals to users, providing them with useful information in real time.

[0773] Input: household accounting database, advertising database

[0774] Output: Push notification of deals

[0775] By following these steps, users can automatically manage physical receipts and electronic receipts from online stores, and efficiently update and use their household accounts.

[0776] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0777] MODE FOR CARRYING OUT THE INVENTION

[0778] The present invention is a system that automatically generates a household account book when a user simply takes a photo of a receipt, and then uses an emotion engine to recognize the user's emotions, manages household finances, and provides information on special offers. The following is a specific example of how the present invention can be implemented.

[0779] System Overview

[0780] The system includes a camera, a receiver, a text information extraction unit, an analysis unit, a database registration unit, a display unit, a push notification unit, and an emotion engine. The system is designed to allow users to easily manage their household finances and efficiently manage and save money. Furthermore, the system recognizes the user's emotions and adjusts the system's behavior and notification content accordingly, improving ease of use.

[0781] Explanation of program processing

[0782] Photograph and send receipt

[0783] User

[0784] The user launches the household account book mini-app and takes a picture of the receipt using the camera function, which then sends the image to the server.

[0785] Receipt image reception and analysis

[0786] server

[0787] The server receives the received receipt image. Next, it uses OCR technology to extract text information from the receipt image. The extracted text information is then analyzed and converted into detailed information such as the store name, product name, price, and purchase date and time.

[0788] Classification and storage of generated data

[0789] server

[0790] The server categorizes the analyzed information. For example, it automatically sorts items into categories such as groceries, daily necessities, and beverages. The sorted data is then saved in a household accounting database.

[0791] Update and view household accounts

[0792] Terminal

[0793] The device periodically retrieves updated household accounting information from the server, and any new data is automatically reflected in the user's app.

[0794] Accumulation and analysis of purchase history

[0795] server

[0796] The server analyzes the purchase history stored in the household accounting database and generates data such as the purchase frequency of specific stores and products.

[0797] Providing discount information

[0798] server

[0799] The server compares the household accounting database with the advertising database and provides relevant discount information to users via push notifications, which can be used to help users save money efficiently.

[0800] Emotion Engine Operation

[0801] server

[0802] The server runs an emotion engine based on the user's facial and voice data to recognize the user's emotions. For example, it uses a camera and microphone to analyze facial expressions and tone of voice to identify emotions such as joy, surprise, sadness, and anger.

[0803] Adjustments to household account display and notifications

[0804] server

[0805] The server adjusts the visualization of the household accounting data and the content of push notifications based on the user's emotions recognized by the emotion engine. For example, if the user is feeling stressed, it will notify them with an encouraging message or a simple task.

[0806] Specific examples

[0807] For example, when a user goes shopping at a supermarket and receives a receipt, the following sequence of events takes place: The user launches the mini-app and takes a photo of the receipt. The image is sent to the server, where OCR technology extracts data such as "Supermarket X, Tomatoes, 200 yen, October 5, 2023." The analysis tool categorizes this data into categories (e.g., "Groceries") and stores it in a database. The user can check the latest household accounting information displayed on the device and understand spending by category and overall spending. Additionally, if the emotion engine detects stress in the user, an interface is displayed that allows them to easily check their spending for the day, and relevant coupons for the next day are sent via push notification.

[0808] The above is an embodiment of the present invention, which allows users to effectively and efficiently manage their finances and receive support tailored to their emotional state.

[0809] The processing flow will be explained below.

[0810] Step 1:

[0811] User

[0812] The user launches the household account book mini-app and selects the camera function. They take a picture of the receipt with the camera and press the capture button. A confirmation screen is then displayed, and the user confirms the image and presses the send button.

[0813] Step 2:

[0814] Terminal

[0815] The device then sends the captured receipt image to the server, where it is encrypted to ensure data security.

[0816] Step 3:

[0817] server

[0818] The server receives the receipt image sent by the user using a receiving means. Once receipt is confirmed, the image is converted into text information using OCR (Optical Character Recognition) technology.

[0819] Step 4:

[0820] server

[0821] The character information extracted by OCR technology is processed by an analysis means to generate detailed information such as the store name, product name, price, purchase date, etc. At this stage, recognition errors are also checked and corrected.

[0822] Step 5:

[0823] server

[0824] The analyzed information is classified by category. For example, it is automatically sorted into categories such as groceries, daily necessities, and beverages. The sorted data is saved in a household accounting database.

[0825] Step 6:

[0826] Terminal

[0827] The device periodically retrieves updated household accounting information from the server, and any new data is automatically reflected in the user's app.

[0828] Step 7:

[0829] User

[0830] Users can check their latest household accounting information within the app and understand their income and expenditure situation. Expenses by category and overall spending status are displayed in graph and list format.

[0831] Step 8:

[0832] server

[0833] The server runs an emotion engine based on the user's facial and voice data to recognize the user's emotions. For example, it uses a camera and microphone to analyze facial expressions and tone of voice to identify emotions such as joy, surprise, sadness, and anger.

[0834] Step 9:

[0835] server

[0836] The emotion engine adjusts the visualization of household accounting data based on the user's emotions. If the user is feeling stressed, the data display will be simplified and the colors will be softened.

[0837] Step 10:

[0838] server

[0839] The system compares the household accounting database with the advertising database to identify deals relevant to the user. For example, if a frequently purchased item is on sale, that information is extracted.

[0840] Step 11:

[0841] Terminal

[0842] The device receives push notifications from the server and displays them to the user, who can view them within the app and can find specific coupons or sales information.

[0843] Step 12:

[0844] User

[0845] Users can check the notifications and take advantage of the displayed deals, making it easy to save money by using coupons and plan their shopping based on the sale information.

[0846] Step 13:

[0847] server

[0848] The emotion engine periodically analyzes the user's emotions and adjusts the content and timing of push notifications. For example, if the user is feeling stressed, it will provide encouraging messages or information to help them relax.

[0849] The above is a specific embodiment of the present invention that combines an emotion engine, allowing users to effectively manage their finances and receive support according to their emotional state.

[0850] Example 2

[0851] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0852] Modern society demands tools that allow individuals to efficiently and effectively manage their household finances. It is particularly important to easily record and classify daily shopping expenses and use them to predict future spending. Furthermore, providing support that takes into account the user's emotional state would improve user satisfaction and ease of use. However, current household finance management systems rarely meet all of these requirements, and they often place a heavy burden on users. Therefore, the challenge is to provide a system that makes it easy for users to manage their spending and provides the necessary information while taking their emotions into account.

[0853] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0854] In this invention, the server includes: a photographing means for a user to photograph a receipt; a receiving means for receiving the receipt image photographed by the photographing means; a character information extraction means for extracting character information from the receipt image using OCR technology; an analysis means for analyzing the character information extracted by the character information extraction means and generating a store name, product name, price, and purchase date and time; a database registration means for categorizing the information generated by the analysis means and storing it in a database; a display means for displaying the information stored in the database; a push notification means for comparing the database with other information databases and providing highly relevant information via push notification; an emotion recognition means for analyzing the user's facial expressions and voice data to recognize emotions; and an adjustment means for adjusting the display content of the display means and the notification content of the push notification means based on the emotion recognized by the emotion recognition means. This not only allows users to easily record and manage their expenses, but also provides them with optimal information according to their emotional state, making household management more efficient and effective.

[0855] "Photographing means" refers to a function or device that allows a user to photograph a receipt with a camera.

[0856] The "receiving means" refers to a function or device for sending image data of a photographed receipt to a server and receiving it.

[0857] "Text information extraction means" refers to the functions and techniques for extracting text information from receipt images using OCR technology.

[0858] The "analysis means" refers to a function or technology for analyzing the character information acquired by the character information extraction means and converting it into detailed data such as the store name, product name, price, and purchase date and time.

[0859] The "database registration means" refers to the function or technology for classifying the information generated by the analysis means into categories and storing it in a database.

[0860] "Display means" refers to functions and technologies for visually presenting information stored in a database to a user.

[0861] "Push notification means" refers to a function or technology that compares the household accounting database with other information databases and provides relevant information to the user via push notification.

[0862] "Emotion recognition means" refers to functions and technologies for analyzing a user's facial expressions and voice data and recognizing the user's emotions.

[0863] The "adjustment means" refers to a function or technology for adjusting the display content of the display means or the notification content of the push notification means based on the emotion recognized by the emotion recognition means.

[0864] MODE FOR CARRYING OUT THE INVENTION

[0865] The present invention is a system that automatically generates a household account book simply by a user taking a photo of a receipt, and then uses an emotion engine to recognize the user's emotions, providing household management and related information. Specific means for implementing the present invention and examples of its operation are shown below.

[0866] System Overview

[0867] The system includes a camera, a receiver, a text information extractor, an analyzer, a database registerer, a displayer, a push notification system, and an emotion recognition system. The system is designed to allow users to easily manage their household finances and achieve efficient spending management and savings. Furthermore, the system's usability is enhanced by recognizing the user's emotions and adjusting the system's behavior and notification content accordingly.

[0868] Photograph and send receipt

[0869] User

[0870] The user launches the household accounting mini-app on their smartphone or tablet and takes a picture of the receipt using the device's camera. For example, they can press the "take a picture of receipt" button on the app's home screen to switch to the camera screen. When the user focuses on the receipt and presses the shutter button, the captured image is sent to the server.

[0871] Receipt image reception and analysis

[0872] server

[0873] The server receives the receipt image sent by the user. It then uses OCR technology such as Tesseract to extract text information from the image. For example, a string of characters in the format "Super X, Tomato, ¥200, October 5, 2023" is generated. This text information is then broken down into detailed data for the database using an analysis tool.

[0874] Classification and storage of generated data

[0875] server

[0876] The server classifies the analyzed data into categories such as "grocery" and "daily necessities." For example, data classified as "grocery" is stored in a MySQL database.

[0877] Update and view household accounts

[0878] Terminal

[0879] The device periodically sends a request to the server to retrieve updated information. If new data is available, it is automatically reflected in the on-screen household ledger display, allowing the user to view it in real time.

[0880] Accumulation and analysis of purchase history

[0881] server

[0882] The server collects past purchase data and analyzes it based on specific items (e.g., by store, frequency of product purchases, etc.). The results of this analysis visualize the purchasing trends of each user.

[0883] Providing discount information

[0884] server

[0885] The server compares the household account data with the advertising database to obtain information such as "Products at a specific store are 10% off this week." This information is then sent to the user as a push notification.

[0886] Operation of emotion recognition means

[0887] server

[0888] The server analyzes the user's facial expressions and voice data in real time to recognize emotions. Using a deep learning model, the user's emotional state is classified into categories such as "surprise," "sadness," and "happiness."

[0889] Adjustments to household account display and notifications

[0890] server

[0891] If the server determines that the user is feeling stressed based on the analysis results of the emotion recognition means, it will provide a visually easy-to-read interface and send push notifications with information on benefits for relaxation.

[0892] Specific examples

[0893] For example, when a user goes shopping at a supermarket and receives a receipt, the following process takes place: The user launches the household accounting mini-app and takes a photo of the receipt. The image is sent to the server, and OCR technology extracts the data: "Supermarket X, Tomatoes, 200 yen, October 5, 2023." The analysis means classifies this data into categories (e.g., "Groceries") and stores it in a database. The user can check the latest household accounting information on their device and understand their spending status by category and overall. Furthermore, if the emotion recognition means detects that the user is stressed, an interface is provided that allows them to easily check their spending status for that day, and relevant coupons for the next day are pushed to them.

[0894] Prompt Sentence Examples

[0895] "Please explain the process of a system that analyzes receipt images taken by users and automatically updates the household account book."

[0896] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0897] Step 1:

[0898] Photograph and send receipt

[0899] The user launches the household account book mini-app and clicks the "Take a photo of receipt" button on the home screen, which launches the camera screen.

[0900] The user focuses on the receipt and presses the shutter button to take a photo of the receipt.

[0901] Input: A receipt image captured by the camera.

[0902] After confirming the image, the user presses the "send" button, and the captured image is sent to the server.

[0903] Output: Receipt image data sent to the server.

[0904] Step 2:

[0905] Receipt image reception and analysis

[0906] The server receives the receipt image sent by the user.

[0907] Input: Receipt image data sent from the user device.

[0908] The server uses OCR technology (e.g., Tesseract OCR) to extract text information from the receipt image.

[0909] Data processing: The process of extracting text information from receipt image data.

[0910] The server analyzes the extracted text information and converts it into detailed data such as the store name, product name, price, and purchase date and time.

[0911] Output: Text information such as store name, product name, price, purchase date and time.

[0912] Step 3:

[0913] Classification and storage of generated data

[0914] The server categorizes the analyzed text information into categories, such as "foodstuffs" and "daily necessities."

[0915] Input: Analyzed text information (store name, product name, price, purchase date and time, etc.).

[0916] The server stores this information in a database (e.g., a MySQL database).

[0917] Data processing: The process of classifying data by category and storing it in a database.

[0918] Output: Database storage of character information categorized by category.

[0919] Step 4:

[0920] Update and view household accounts

[0921] The device periodically sends requests to the server to obtain updated household accounting information.

[0922] Input: Updated household accounting information provided by the server.

[0923] If the device has new data, it will automatically be reflected in the household ledger display on the screen.

[0924] Data processing: Display processing of new household accounting data.

[0925] Output: The updated household accounting information is displayed on the user's device.

[0926] Step 5:

[0927] Accumulation and analysis of purchase history

[0928] The server collects the purchase history stored in the household account book database and performs analysis based on specific items (e.g., by store, frequency of purchase of product, etc.).

[0929] Input: Past purchase history stored in the household accounting database.

[0930] The server aggregates this data and visualizes each user's purchasing trends.

[0931] Data processing: Analysis and aggregation of purchase history data.

[0932] Output: Purchasing trend data based on the analysis results.

[0933] Step 6:

[0934] Providing discount information

[0935] The server compares the household account book database with other information databases to obtain discount information relevant to the user.

[0936] Input: Information from the household accounting database and advertising database.

[0937] The server provides relevant information to the user via push notifications.

[0938] Data processing: Cross-database information matching and notification content generation.

[0939] Output: Deals sent via push notification.

[0940] Step 7:

[0941] Operation of emotion recognition means

[0942] The user allows access to the device's camera and microphone.

[0943] The server uses emotion recognition means to analyze the user's facial expressions and voice data and recognize emotions.

[0944] Input: User's facial expression data and voice data.

[0945] The server uses a deep learning model to identify the emotional state.

[0946] Data processing: Analysis and processing of facial expression and voice data.

[0947] Output: Identified emotional state data.

[0948] Step 8:

[0949] Adjustments to household account display and notifications

[0950] The server adjusts the visualization method of the household accounting data and the notification content based on the results of the emotion recognition means.

[0951] Input: Identified emotional state data and household ledger information.

[0952] The server generates an interface and notification content that matches the user's emotions.

[0953] Data processing: Adjustment of display and notification content based on emotional state.

[0954] Output: Adjusted interface and push notification content.

[0955] (Application example 2)

[0956] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0957] In conventional household management systems, entering expenditure data is tedious, and it is particularly difficult to fill out receipts while driving or on the go. Furthermore, the systems do not take the user's feelings into consideration, resulting in poor usability. Furthermore, they do not adequately provide advertisements or discount information linked to receipt information, making it difficult for users to save money efficiently. To solve these problems, it is necessary to improve the user experience in household management systems.

[0958] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0959] In this invention, the server includes a photographing means for a user to photograph a receipt, a receiving means for receiving the receipt image photographed by the photographing means, a character information extraction means for extracting character information from the receipt image using OCR technology, an analysis means for analyzing the character information extracted by the character information extraction means and generating a store name, product name, price, and purchase date and time, a database registration means for categorizing the information generated by the analysis means and storing it in a household accounting database, a display means for displaying the information stored in the household accounting database, a push notification means for comparing the household accounting database with an advertising database and providing relevant discount information via push notification, an emotion recognition means for recognizing the user's emotion, an adjustment means for adjusting the display method of the household accounting data and the notification content based on the emotion analyzed by the emotion recognition means, and an integration means integrated into the infotainment system of an autonomous vehicle and coordinating with an in-vehicle camera and a voice assistant. This enables safe household management while driving and provides optimal information tailored to the user's emotion.

[0960] A "photographing means" is a device used by a user to capture an image of a printed medium such as a receipt.

[0961] The "receiving means" is a function for importing the receipt image captured by the image capturing means into the system.

[0962] The "character information extraction means" is a function that uses OCR technology to recognize and extract characters from a captured receipt image.

[0963] The "analysis means" is a function that analyzes the extracted character information and converts it into specific data such as the store name, product name, price, and purchase date and time.

[0964] The "database registration means" is a function for classifying the information generated by the analysis means into categories and storing the information in the household account book database.

[0965] The "display means" is a device or function for visually presenting the information stored in the household accounting database to the user.

[0966] The "push notification means" is a function that checks the household account book database against the advertisement database and notifies the user of relevant deals.

[0967] The "emotion recognition means" is a function for analyzing and recognizing emotions from the user's voice, facial expressions, etc.

[0968] The "adjustment means" is a function that adjusts the display method of household accounting data and notification content based on the user's emotions analyzed by the emotion recognition means.

[0969] "Integration means" refers to the functionality for integrating and linking other systems and devices with the infotainment system of an autonomous vehicle.

[0970] The present invention is a system that is integrated into the infotainment system of an autonomous vehicle, allowing users to safely and efficiently manage their household finances while driving. The system mainly includes a photographing means, a receiving means, a text information extraction means, an analysis means, a database registration means, a display means, a push notification means, an emotion recognition means, an adjustment means, and an integration means.

[0971] Hardware and Software Used

[0972] Hardware:

[0973] In-car camera

[0974] microphone

[0975] Touch panel display

[0976] In-vehicle computer

[0977] software:

[0978] OCR engine (e.g. Tesseract)

[0979] Emotion recognition engine (e.g. Microsoft Azure Cognitive Services)

[0980] Speech synthesis engine (e.g. Google Text-to-Speech API)

[0981] Cloud databases (e.g. Firebase)

[0982] Infotainment System API

[0983] System Operation

[0984] User

[0985] After shopping, users can take a photo of the receipt with the car's camera, ensuring safety while driving.

[0986] server

[0987] 1. Receipt reception and OCR processing:

[0988] The receiving device captures the receipt image taken by the in-car camera, and the OCR engine extracts text information from the image. This process generates data such as the store name, product name, price, and purchase date and time.

[0989] 2. Data analysis and classification:

[0990] The analysis means analyzes various data using the extracted character information, and the database registration means stores the information in the household account book database. At this time, the data is automatically classified into categories such as food, daily necessities, and beverages.

[0991] 3. Matching purchase history with advertising data:

[0992] The server compares the household account database with the advertisement database to generate relevant deals for the user, which are then provided to the user via push notification.

[0993] 4. Emotion recognition:

[0994] An emotion recognition engine analyzes voice data collected through microphones in the car, thereby identifying the user's current emotion (happiness, surprise, sadness, anger, etc.).

[0995] 5. Notification Adjustments:

[0996] The application adjusts the display method of household accounting data and notification content based on the emotion information acquired by the emotion recognition means, thereby providing optimal information suited to the user.

[0997] Terminal

[0998] The device (touch panel display of the infotainment system) periodically retrieves the latest household accounting information sent from the server and visually displays it. It also has a function to read out the contents of push notifications using a speech synthesis engine.

[0999] Specific examples

[1000] For example, if a user is driving an autonomous vehicle and purchases coffee at a drive-thru and receives a receipt, they simply present the receipt to the in-car camera. The onboard computer uses an OCR engine to extract information such as "coffee shop, coffee, 400 yen, October 10, 2023." The data is then stored and categorized in a Firebase cloud database. The infotainment system's touchscreen display displays the latest household finances and provides relevant deals via voice. If the emotion recognition engine determines the user is stressed, the system will notify them with a gentle voice message encouraging them to relax.

[1001] Prompt Sentence Examples

[1002] When a user takes a photo of a receipt with the in-car camera, the OCR engine extracts the text information and stores it in Firebase. The emotion recognition engine analyzes the user's voice and identifies their emotion. Next, the household accounting information is displayed on the touch panel, and if necessary, the speech synthesis engine provides discount information.

[1003] This system allows users to intuitively and safely manage their household finances while driving, and provides optimal information based on their emotional state. This technology is specialized for autonomous vehicles, contributing to an improved user experience.

[1004] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1005] Step 1:

[1006] The user takes a photo of the receipt

[1007] After shopping, the user presents the receipt to the camera in the autonomous vehicle and takes a photo. The input is the receipt image, and the output is the transfer of image data to the on-board computer. User interaction in this step is kept to a minimum.

[1008] Step 2:

[1009] Receiving receipt images

[1010] The server (on-board computer) receives receipt images taken by the user with the in-car camera. The input is the receipt image, and the output is image data provided to the OCR engine. This step ensures that the receipt image is captured in the system.

[1011] Step 3:

[1012] OCR analysis

[1013] Extract text information from the receipt image using an OCR engine (e.g., Tesseract). The input is the receipt image, and the output is the extracted text information (store name, product name, price, purchase date and time). This step converts the image data into text data.

[1014] Step 4:

[1015] Data analysis

[1016] The server analyzes the extracted text and generates the appropriate information (store name, product name, price, purchase date and time). The input is text, and the output is the analyzed data. This step involves formatting and categorizing the data.

[1017] Step 5:

[1018] Saving to a database

[1019] The database registration means classifies the analyzed information by category and stores it in the household account book database. The input is the analyzed data, and the output is the database registration of the classified data. This allows the user to check their expenses by category.

[1020] Step 6:

[1021] Displaying household accounts

[1022] The terminal (the display of the in-vehicle infotainment system) periodically retrieves the latest household accounting information from the server and displays it. The input is information from the household accounting database, and the output is a visual presentation of the information on the display. At this step, the user can check the expenditure information.

[1023] Step 7:

[1024] Matching purchase history with advertising data

[1025] The server compares the household account database with the advertisement database and generates deals related to the user. The input is household account data and advertisement data, and the output is deals. This step generates information useful to the user.

[1026] Step 8:

[1027] Push notifications

[1028] The push notification means notifies the user of the discount information generated in the previous step. The input is the discount information, and the output is a real-time notification to the user. In this step, the user receives the information efficiently.

[1029] Step 9:

[1030] emotion recognition

[1031] The emotion recognition means analyzes the voice data collected using the in-car microphone and identifies the user's emotion. The input is voice data, and the output is the user's emotional information. This step identifies the user's emotional state.

[1032] Step 10:

[1033] Adjusting notification content

[1034] The server adjusts the display method of the household accounting data and the notification content based on the emotional information analyzed by the emotion recognition means. The input is the user's emotional information, and the output is the adjusted display method and notification content. This step provides the user with the most appropriate information.

[1035] The above is a detailed description of each processing step, which allows the system to provide advanced household management and emotion-based services while ensuring user safety.

[1036] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1037] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1038] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1039] [Third embodiment]

[1040] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1041] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1042] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1043] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1044] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1045] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1046] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1047] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1048] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1049] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1050] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1051] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1052] MODE FOR CARRYING OUT THE INVENTION

[1053] The present invention provides a system that automatically generates a household account book by allowing a user to simply take a photo of a receipt. The following is a specific example for implementing the present invention.

[1054] System Overview

[1055] The purpose of this system is to enable users to manage their household finances hassle-free. The system includes a photographing means, a receiving means, a text information extraction means, an analysis means, a database registration means, a display means, and a push notification means. This allows users to easily grasp their spending situation, obtain future spending forecasts, and obtain advantageous information.

[1056] Explanation of program processing

[1057] Photograph and send receipt

[1058] User

[1059] The user launches the household account book mini-app and takes a picture of the receipt using the camera function, which then sends the image to the server.

[1060] Receipt image reception and analysis

[1061] server

[1062] The server receives the received receipt image. Next, it uses OCR technology to extract text information from the receipt image. The extracted text information is then analyzed and converted into detailed information such as the store name, product name, price, and purchase date and time.

[1063] Classification and storage of generated data

[1064] server

[1065] The server categorizes the analyzed information and stores it in a household accounting database, which is continuously updated to accumulate the user's purchasing history.

[1066] Update and view household accounts

[1067] Terminal

[1068] The terminal receives the latest household accounting information from the server and displays it to the user, allowing the user to easily manage their household finances.

[1069] Accumulation and analysis of purchase history

[1070] server

[1071] The server analyzes the purchase history stored in the household accounting database and generates data such as the purchase frequency of specific stores and products.

[1072] Providing discount information

[1073] server

[1074] The server compares the household accounting database with the advertising database and delivers relevant information via push notifications, which users can use to save money efficiently.

[1075] Specific examples

[1076] For example, if a user goes shopping at a supermarket and receives a receipt, the following happens: The user launches the mini-app and takes a photo of the receipt. The image is sent to the server, and OCR technology extracts data such as "Supermarket X, Tomatoes, 200 yen, October 5, 2023." The analysis means categorizes this data into categories (e.g., "Groceries") and stores it in a database. The user's device displays the data the next day and is also notified that Supermarket X's tomatoes will be on sale over the weekend.

[1077] The above is an embodiment of the present invention, which allows the user to manage their household finances effectively and efficiently.

[1078] The processing flow will be explained below.

[1079] Step 1:

[1080] User

[1081] The user launches the household account book mini-app and selects the camera function. They take a picture of the receipt with the camera and press the capture button. A confirmation screen is then displayed, and the user confirms the image and presses the send button.

[1082] Step 2:

[1083] Terminal

[1084] The device then sends the captured receipt image to the server, where it is encrypted to ensure data security.

[1085] Step 3:

[1086] server

[1087] The server receives the receipt image sent by the user using a receiving means. Once receipt is confirmed, the image is converted into text information using OCR (Optical Character Recognition) technology.

[1088] Step 4:

[1089] server

[1090] The character information extracted by OCR technology is processed by an analysis means to generate detailed information such as the store name, product name, price, purchase date, etc. At this stage, recognition errors are also checked and corrected.

[1091] Step 5:

[1092] server

[1093] The analyzed information is classified by category. For example, it is automatically sorted into categories such as groceries, daily necessities, and beverages. The sorted data is saved in a household accounting database.

[1094] Step 6:

[1095] Terminal

[1096] The device periodically retrieves updated household accounting information from the server, and any new data is automatically reflected in the user's app.

[1097] Step 7:

[1098] User

[1099] Users can check their latest household accounting information within the app and understand their income and expenditure situation. Expenses by category and overall spending status are displayed in graph and list format.

[1100] Step 8:

[1101] server

[1102] The system analyzes a user's purchasing history based on data from the previous month and past data, generating analysis results such as purchase frequency of specific stores and products, and monthly spending trends.

[1103] Step 9:

[1104] server

[1105] The system compares the household accounting database with the advertising database to identify deals relevant to the user. For example, if a frequently purchased item is on sale, that information is extracted.

[1106] Step 10:

[1107] Terminal

[1108] The device receives push notifications from the server and displays them to the user, who can view them within the app and can find specific coupons or sales information.

[1109] Step 11:

[1110] User

[1111] Users can check the notifications and take advantage of the displayed deals, making it easy to save money by using coupons and plan their shopping based on the sale information.

[1112] Example 1

[1113] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1114] Traditional household management methods are time-consuming and require users to manually input data, resulting in problems with accuracy and efficiency. Furthermore, few systems offer added value, such as predicting future spending based on purchase history or providing information on special offers. This makes it difficult for users to effectively manage their household finances and budgets.

[1115] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1116] In this invention, the server includes an image capture means for a user to take a picture of a receipt, an image receiving means for receiving the receipt image captured by the image capture means, a character information extraction means for extracting character information from the receipt image using optical character recognition technology, an information analysis means for analyzing the character information extracted by the character information extraction means and generating a store name, product name, price, and purchase date and time, a database recording means for categorizing the information generated by the information analysis means and storing it in a financial management database, an information display means for displaying the information stored in the financial management database, and a notification means for comparing the financial management database with an advertising database and providing relevant discount information via push notification. This allows users to efficiently manage their household finances simply by taking pictures of receipts, and also enables them to receive future spending forecasts and special offers based on their purchase history.

[1117] "Image capture means" refers to a device or method that allows a user to take a photo of a receipt.

[1118] "Image receiving means" refers to a device or method for receiving a captured receipt image from the user's terminal.

[1119] Optical character recognition (OCR) is a technology that extracts text information from an image.

[1120] "Text information extraction means" refers to a device or method for extracting text information from a receipt image.

[1121] The "information analysis means" refers to a device or method that analyzes the extracted character information and generates detailed information such as the store name, product name, price, and purchase date and time.

[1122] The "database recording means" refers to a device or method for categorizing the analyzed information and storing it in a financial management database.

[1123] "Information display means" refers to a device or method for displaying information stored in the financial management database to a user.

[1124] The "notification means" refers to a device or method that cross-references the financial management database with the advertising database and provides relevant discount information to the user via push notification.

[1125] The present invention is a system that automatically generates a household account book by allowing a user to simply take a photo of a receipt. The following is a specific example for implementing the present invention.

[1126] System Overview

[1127] The purpose of this system is to allow users to manage their household finances hassle-free. The system includes the following components:

[1128] 1. Image acquisition method

[1129] 2. Image Receiving Method

[1130] 3. Optical character recognition technology (OCR)

[1131] 4. Text information extraction method

[1132] 5. Information analysis means

[1133] 6. Database Recording Method

[1134] 7. Information display means

[1135] 8. Means of notification

[1136] Photograph and send receipt

[1137] User

[1138] The user launches the dedicated household accounting mini-app. Next, they take a photo of the receipt using the smartphone's camera. This involves the user operating the camera and adjusting it so that the entire receipt is captured clearly. Once the photo is taken, the app provides instructions for sending the image to the server. The user presses the send button to send the image to the server.

[1139] Receipt image reception and analysis

[1140] server

[1141] The server receives the receipt image sent from the user's device. This reception process includes a method to ensure security using the HTTPS protocol. Next, the server uses the Tesseract OCR library to extract text information from the received image. This OCR process generates text data, which provides detailed information such as the store name, product name, price, and purchase date and time. Furthermore, the extracted text information is structured and organized by category using information analysis means.

[1142] Classification and storage of generated data

[1143] server

[1144] The server classifies the data generated by the information analysis means into categories, such as food, beverages, and daily necessities. This classified data is stored in a financial management database using a database recording means. This database uses a relational database management system such as MySQL or PostgreSQL. The data is continuously updated, and user purchase histories are accumulated.

[1145] Update and view household accounts

[1146] Terminal

[1147] The device periodically accesses the server to obtain the latest household accounting data. Data is obtained via communication via a RESTful API. The received data is temporarily stored in the device's local storage and displayed on a dedicated UI (user interface). This display process uses UI libraries such as React Native. On this screen, the user can view a list of expenses by day and category.

[1148] Accumulation and analysis of purchase history

[1149] server

[1150] The server periodically analyzes the purchase history data stored in the database. This analysis is performed using Python data processing libraries (Pandas and NumPy). For example, graphs are created to show the purchase frequency of specific products and trends in total spending. The results of this analysis are used to generate individual reports for each user.

[1151] Providing discount information

[1152] server

[1153] The server compares the financial management database with the advertising database to generate relevant discount information. This process uses ad distribution services such as the AdSense API. The generated discount information is then provided to users via push notifications. Firebase Cloud Messaging (FCM) is used for push notifications. For example, the server can notify users in real time of sale information for products they frequently purchase.

[1154] Specific examples

[1155] For example, consider the case where a user goes shopping at a supermarket and receives a receipt. The user launches the household accounting mini-app and takes a picture of the receipt using the smartphone's camera. The image is sent to the server, and OCR technology extracts text information such as "Supermarket X, Tomatoes, 200 yen, October 5, 2023." Analysis means categorizes this data into the "grocery" category and stores it in a database. The next day, the user's device receives the latest household accounting data and displays it on the screen. Additionally, a push notification is sent informing them that Supermarket X tomatoes will be on sale over the weekend.

[1156] Example prompts to input to the generative AI model

[1157] Using prompts like the following can effectively leverage generative AI models (e.g., GPT-4):

[1158] "Please explain the automatic household accounting system. The user takes a photo of a receipt and sends the image to a server. The server uses OCR technology to extract text information from the receipt, analyzes it, and saves it in a database. The device receives and displays the latest household accounting data, and the server analyzes purchase history and provides savings information."

[1159] The above is a specific embodiment for carrying out the present invention. This system allows the user to manage their household finances effectively and efficiently.

[1160] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1161] Step 1:

[1162] User

[1163] The user launches the household account book mini-app and takes a photo of the receipt using the smartphone's camera. The user adjusts the camera so that the entire receipt is clearly visible, and then presses the "take a photo" button.

[1164] Input: Receipt image taken by the camera via user operation

[1165] Output: Photographed receipt image data

[1166] Step 2:

[1167] User

[1168] The user presses the "Send" button in the app to send the captured image of the receipt to the server. The app encodes the image data and sends it to the server using the HTTPS protocol.

[1169] Input: Photographed receipt image data

[1170] Output: Receipt image data sent to the server

[1171] Step 3:

[1172] server

[1173] The server receives the sent receipt image. This reception uses the HTTPS protocol to ensure security. The received image data is temporarily stored in server storage.

[1174] Input: Receipt image data sent by the user

[1175] Output: Receipt image data stored on the server

[1176] Step 4:

[1177] server

[1178] The server calls the Tesseract OCR library to extract text information from the received receipt image. The OCR process generates the receipt's text data.

[1179] Input: Receipt image data stored on the server

[1180] Output: Extracted text information (store name, product name, price, purchase date and time, etc.)

[1181] Step 5:

[1182] server

[1183] The server analyzes the text information to identify the store name, product name, price, purchase date, etc. It also categorizes the information into categories, such as food, beverages, and daily necessities.

[1184] Input: Extracted text information

[1185] Output: Parsed and categorized information

[1186] Step 6:

[1187] server

[1188] The server stores the classified information in a financial management database, which also checks for duplicate data. The database uses a relational database management system (MySQL or PostgreSQL).

[1189] Input: Parsed and classified information

[1190] Output: Purchase history information stored in the database

[1191] Step 7:

[1192] Terminal

[1193] The device periodically accesses the server to retrieve the latest household accounting data. This process uses communication via a RESTful API. The retrieved data is temporarily stored in local storage and displayed in a dedicated UI.

[1194] Input: Latest household accounting data obtained from the server

[1195] Output: Household accounting information displayed on the terminal

[1196] Step 8:

[1197] server

[1198] The server periodically analyzes the purchase history data stored in the financial management database. This analysis uses Python data processing libraries (Pandas and NumPy) to generate data such as the purchase frequency of specific products and trends in total spending.

[1199] Input: Purchase history data stored in the financial management database

[1200] Output: Analysis results (purchase frequency, trends in total expenditure, etc.)

[1201] Step 9:

[1202] server

[1203] The server compares the financial management database with the advertising database to generate relevant discount information. This is done using an ad distribution service such as the AdSense API. The generated discount information is then sent to users via push notifications.

[1204] Input: Analysis results, advertising database information

[1205] Output: A push notification with the discount information sent to the user.

[1206] This allows users to efficiently manage their household finances simply by taking photos of receipts, and they can also receive future spending forecasts and special offers based on their purchasing history.

[1207] (Application example 1)

[1208] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1209] In recent years, with the spread of e-commerce, an increasing number of users use multiple online shopping sites. However, manually managing the electronic receipts received from each online shopping site is cumbersome, resulting in a decrease in the efficiency of users' household finance management. In addition, it is difficult to integrate and manage online and offline purchase data, making it difficult to efficiently forecast expenses and manage budgets. This increases the burden on users when managing their household finances.

[1210] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1211] In this invention, the server includes a photographing device for a user to photograph a receipt, a receiving device for receiving the receipt image photographed by the photographing device, a character information extraction device for extracting character information from the receipt image using OCR technology, an analysis device for analyzing the character information extracted by the character information extraction device and generating a store name, product name, price, and purchase date and time, a database registration device for categorizing the information generated by the analysis device and storing it in a household accounting database, a display device for displaying the information stored in the household accounting database, a push notification device for comparing the household accounting database with an advertising database and providing relevant discount information via push notification, and a data acquisition device for automatically acquiring and analyzing electronic receipts from online stores. This allows users to automatically acquire, analyze, and manage not only physical receipts but also electronic receipt information from online stores. This not only enables more efficient household management but also improves users' overall spending forecasting and budget management.

[1212] "Photographing means" refers to the device or function that a user uses to photograph a physical receipt.

[1213] The "receiving means" refers to a device or function for transmitting the receipt image captured by the capturing means to the server and receiving it.

[1214] "Text information extraction means" refers to functions or software for extracting text information from receipt images using OCR technology.

[1215] "Analysis means" refers to a function or software that analyzes the extracted character information and generates detailed information such as the store name, product name, price, and purchase date and time.

[1216] The "database registration means" refers to a function or software that classifies the information generated by the analysis means into categories and stores it in the household accounting database.

[1217] "Display means" refers to devices or software that visually present the information stored in the household accounting database to the user.

[1218] The "push notification means" is a function that compares the household accounting database with the advertisement database and provides relevant discount information to the user as a push notification.

[1219] "Data acquisition means" refers to functions or software for automatically acquiring and analyzing electronic receipts from online stores.

[1220] The present invention provides a system that automatically generates a household account book by allowing users to simply take a photo of a receipt. In particular, the present invention has the ability to automatically acquire and analyze electronic receipts from online stores. The following are specific embodiments for carrying out the present invention.

[1221] System Overview

[1222] The purpose of this system is to enable users to manage their household finances hassle-free. The system includes a photographing means, a receiving means, a text information extraction means, an analysis means, a database registration means, a display means, a push notification means, and a data acquisition means.

[1223] Program processing explanation

[1224] Photograph and send receipt

[1225] The user launches the household accounting app and takes a photo of the physical receipt using the smartphone's camera, which then sends the image to the server.

[1226] Receipt image reception and analysis

[1227] The server receives the receipt image and extracts the text information using an OCR library (Tesseract OCR). The extracted text information is then analyzed and converted into detailed information such as the store name, product name, price, and purchase date and time.

[1228] Classification and storage of generated data

[1229] The server categorizes the analyzed information and stores it in a SQLite database, which is continually updated to accumulate the user's purchasing history.

[1230] Update and view household accounts

[1231] The device receives the latest household accounting information from the server and displays it to the user using React Native, allowing users to easily manage their household finances.

[1232] Automatic capture and analysis of electronic receipts

[1233] The server accesses the user's shopping site account and automatically retrieves the electronic receipt. This process can be set to occur periodically. The retrieved electronic receipt is analyzed using OCR technology to extract text information similar to that of a physical receipt.

[1234] Accumulation and analysis of purchase history

[1235] The server analyzes the purchase history stored in the household account book database and generates data such as the frequency of purchases of specific stores and products, which allows the server to understand users' purchasing patterns and trends.

[1236] Providing discount information

[1237] The server compares the household accounting database with the advertising database and delivers relevant deals via push notifications, such as notifications about sales on specific products.

[1238] Specific examples

[1239] For example, if a user purchases an item from an online store and receives an electronic receipt, the following happens: The server automatically checks the purchase history of the online shopping site periodically and retrieves the electronic receipt, even if the user does not launch the household accounting app. OCR technology extracts information such as "online store, product name, 5000 yen, October 5, 2023," and an analysis method classifies this information into a category (for example, "entertainment") and stores it in a database. When the user opens the app, they can check the latest household accounting information and receive notifications when specific products go on sale.

[1240] Prompt Sentence Examples

[1241] "We will develop an application that analyzes digital receipts received from online stores and reflects them in a household ledger. The process involves obtaining purchase history, extracting text information using OCR technology, storing it in a database, and displaying it in a user interface. The tools used are React Native, Tesseract OCR, and SQLite."

[1242] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1243] Step 1:

[1244] The user launches the household accounting app on their smartphone and takes a photo of the physical receipt. By doing this, the user captures the receipt image using the camera function.

[1245] Input: Physical receipt

[1246] Output: Receipt image

[1247] Step 2:

[1248] The server receives the receipt image sent from the smartphone, and the receipt image is stored on the server as a data packet.

[1249] Input: Receipt image

[1250] Output: Receipt image stored on the server

[1251] Step 3:

[1252] The server uses an OCR library (Tesseract OCR) to extract text information from the received receipt image. During this process, the image data is converted into text data.

[1253] Input: Receipt image

[1254] Output: Extracted text information (store name, product name, price, purchase date and time)

[1255] Step 4:

[1256] The server uses an analysis tool to analyze the extracted text information and generate detailed information such as the store name, product name, price, and purchase date and time, allowing each item to be identified and organized as usable data.

[1257] Input: Extracted text information

[1258] Output: Organized detailed information (store name, product name, price, purchase date and time)

[1259] Step 5:

[1260] The server categorizes the details into categories (e.g., groceries, entertainment) and stores them in an SQLite database. During this process, the data is classified and registered for storage.

[1261] Input: Organized details

[1262] Output: Database records containing information categorized by category

[1263] Step 6:

[1264] The server accesses the user's online shopping site account and automatically retrieves electronic receipts. It periodically checks the purchase history and downloads new electronic receipts.

[1265] Input: User's online shopping site account information

[1266] Output: The obtained e-receipt

[1267] Step 7:

[1268] The server analyzes the acquired electronic receipt using OCR technology to extract text information. Just like physical receipts, text data is generated from electronic receipts.

[1269] Input: Retrieved e-receipt

[1270] Output: Extracted text information (store name, product name, price, purchase date and time)

[1271] Step 8:

[1272] The server categorizes the text information extracted from the electronic receipts and stores it in a database, where it is managed together with the physical receipt data.

[1273] Input: Extracted text information (electronic receipt)

[1274] Output: Database records containing categorized e-receipt information

[1275] Step 9:

[1276] The device receives the latest household accounting data from the server and displays it to the user. React Native is used to provide an intuitive visual interface.

[1277] Input: Latest household accounting data received from the server

[1278] Output: User interface reflecting household accounting data

[1279] Step 10:

[1280] The server compares the household accounting database with the advertising database and pushes relevant deals to users, providing them with useful information in real time.

[1281] Input: household accounting database, advertising database

[1282] Output: Push notification of deals

[1283] By following these steps, users can automatically manage physical receipts and electronic receipts from online stores, and efficiently update and use their household accounts.

[1284] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1285] MODE FOR CARRYING OUT THE INVENTION

[1286] The present invention is a system that automatically generates a household account book when a user simply takes a photo of a receipt, and then uses an emotion engine to recognize the user's emotions, manages household finances, and provides information on special offers. The following is a specific example of how the present invention can be implemented.

[1287] System Overview

[1288] The system includes a camera, a receiver, a text information extraction unit, an analysis unit, a database registration unit, a display unit, a push notification unit, and an emotion engine. The system is designed to allow users to easily manage their household finances and efficiently manage and save money. Furthermore, the system recognizes the user's emotions and adjusts the system's behavior and notification content accordingly, improving ease of use.

[1289] Explanation of program processing

[1290] Photograph and send receipt

[1291] User

[1292] The user launches the household account book mini-app and takes a picture of the receipt using the camera function, which then sends the image to the server.

[1293] Receipt image reception and analysis

[1294] server

[1295] The server receives the received receipt image. Next, it uses OCR technology to extract text information from the receipt image. The extracted text information is then analyzed and converted into detailed information such as the store name, product name, price, and purchase date and time.

[1296] Classification and storage of generated data

[1297] server

[1298] The server categorizes the analyzed information. For example, it automatically sorts items into categories such as groceries, daily necessities, and beverages. The sorted data is then saved in a household accounting database.

[1299] Update and view household accounts

[1300] Terminal

[1301] The device periodically retrieves updated household accounting information from the server, and any new data is automatically reflected in the user's app.

[1302] Accumulation and analysis of purchase history

[1303] server

[1304] The server analyzes the purchase history stored in the household accounting database and generates data such as the purchase frequency of specific stores and products.

[1305] Providing discount information

[1306] server

[1307] The server compares the household accounting database with the advertising database and provides relevant discount information to users via push notifications, which can be used to help users save money efficiently.

[1308] Emotion Engine Operation

[1309] server

[1310] The server runs an emotion engine based on the user's facial and voice data to recognize the user's emotions. For example, it uses a camera and microphone to analyze facial expressions and tone of voice to identify emotions such as joy, surprise, sadness, and anger.

[1311] Adjustments to household account display and notifications

[1312] server

[1313] The server adjusts the visualization of the household accounting data and the content of push notifications based on the user's emotions recognized by the emotion engine. For example, if the user is feeling stressed, it will notify them with an encouraging message or a simple task.

[1314] Specific examples

[1315] For example, when a user goes shopping at a supermarket and receives a receipt, the following sequence of events takes place: The user launches the mini-app and takes a photo of the receipt. The image is sent to the server, where OCR technology extracts data such as "Supermarket X, Tomatoes, 200 yen, October 5, 2023." The analysis tool categorizes this data into categories (e.g., "Groceries") and stores it in a database. The user can check the latest household accounting information displayed on the device and understand spending by category and overall spending. Additionally, if the emotion engine detects stress in the user, an interface is displayed that allows them to easily check their spending for the day, and relevant coupons for the next day are sent via push notification.

[1316] The above is an embodiment of the present invention, which allows users to effectively and efficiently manage their finances and receive support tailored to their emotional state.

[1317] The processing flow will be explained below.

[1318] Step 1:

[1319] User

[1320] The user launches the household account book mini-app and selects the camera function. They take a picture of the receipt with the camera and press the capture button. A confirmation screen is then displayed, and the user confirms the image and presses the send button.

[1321] Step 2:

[1322] Terminal

[1323] The device then sends the captured receipt image to the server, where it is encrypted to ensure data security.

[1324] Step 3:

[1325] server

[1326] The server receives the receipt image sent by the user using a receiving means. Once receipt is confirmed, the image is converted into text information using OCR (Optical Character Recognition) technology.

[1327] Step 4:

[1328] server

[1329] The character information extracted by OCR technology is processed by an analysis means to generate detailed information such as the store name, product name, price, purchase date, etc. At this stage, recognition errors are also checked and corrected.

[1330] Step 5:

[1331] server

[1332] The analyzed information is classified by category. For example, it is automatically sorted into categories such as groceries, daily necessities, and beverages. The sorted data is saved in a household accounting database.

[1333] Step 6:

[1334] Terminal

[1335] The device periodically retrieves updated household accounting information from the server, and any new data is automatically reflected in the user's app.

[1336] Step 7:

[1337] User

[1338] Users can check their latest household accounting information within the app and understand their income and expenditure situation. Expenses by category and overall spending status are displayed in graph and list format.

[1339] Step 8:

[1340] server

[1341] The server runs an emotion engine based on the user's facial and voice data to recognize the user's emotions. For example, it uses a camera and microphone to analyze facial expressions and tone of voice to identify emotions such as joy, surprise, sadness, and anger.

[1342] Step 9:

[1343] server

[1344] The emotion engine adjusts the visualization of household accounting data based on the user's emotions. If the user is feeling stressed, the data display will be simplified and the colors will be softened.

[1345] Step 10:

[1346] server

[1347] The system compares the household accounting database with the advertising database to identify deals relevant to the user. For example, if a frequently purchased item is on sale, that information is extracted.

[1348] Step 11:

[1349] Terminal

[1350] The device receives push notifications from the server and displays them to the user, who can view them within the app and can find specific coupons or sales information.

[1351] Step 12:

[1352] User

[1353] Users can check the notifications and take advantage of the displayed deals, making it easy to save money by using coupons and plan their shopping based on the sale information.

[1354] Step 13:

[1355] server

[1356] The emotion engine periodically analyzes the user's emotions and adjusts the content and timing of push notifications. For example, if the user is feeling stressed, it will provide encouraging messages or information to help them relax.

[1357] The above is a specific embodiment of the present invention that combines an emotion engine, allowing users to effectively manage their finances and receive support according to their emotional state.

[1358] Example 2

[1359] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1360] Modern society demands tools that allow individuals to efficiently and effectively manage their household finances. It is particularly important to easily record and classify daily shopping expenses and use them to predict future spending. Furthermore, providing support that takes into account the user's emotional state would improve user satisfaction and ease of use. However, current household finance management systems rarely meet all of these requirements, and they often place a heavy burden on users. Therefore, the challenge is to provide a system that makes it easy for users to manage their spending and provides the necessary information while taking their emotions into account.

[1361] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1362] In this invention, the server includes: a photographing means for a user to photograph a receipt; a receiving means for receiving the receipt image photographed by the photographing means; a character information extraction means for extracting character information from the receipt image using OCR technology; an analysis means for analyzing the character information extracted by the character information extraction means and generating a store name, product name, price, and purchase date and time; a database registration means for categorizing the information generated by the analysis means and storing it in a database; a display means for displaying the information stored in the database; a push notification means for comparing the database with other information databases and providing highly relevant information via push notification; an emotion recognition means for analyzing the user's facial expressions and voice data to recognize emotions; and an adjustment means for adjusting the display content of the display means and the notification content of the push notification means based on the emotion recognized by the emotion recognition means. This not only allows users to easily record and manage their expenses, but also provides them with optimal information according to their emotional state, making household management more efficient and effective.

[1363] "Photographing means" refers to a function or device that allows a user to photograph a receipt with a camera.

[1364] The "receiving means" refers to a function or device for sending image data of a photographed receipt to a server and receiving it.

[1365] "Text information extraction means" refers to the functions and techniques for extracting text information from receipt images using OCR technology.

[1366] The "analysis means" refers to a function or technology for analyzing the character information acquired by the character information extraction means and converting it into detailed data such as the store name, product name, price, and purchase date and time.

[1367] The "database registration means" refers to the function or technology for classifying the information generated by the analysis means into categories and storing it in a database.

[1368] "Display means" refers to functions and technologies for visually presenting information stored in a database to a user.

[1369] "Push notification means" refers to a function or technology that compares the household accounting database with other information databases and provides relevant information to the user via push notification.

[1370] "Emotion recognition means" refers to functions and technologies for analyzing a user's facial expressions and voice data and recognizing the user's emotions.

[1371] The "adjustment means" refers to a function or technology for adjusting the display content of the display means or the notification content of the push notification means based on the emotion recognized by the emotion recognition means.

[1372] MODE FOR CARRYING OUT THE INVENTION

[1373] The present invention is a system that automatically generates a household account book simply by a user taking a photo of a receipt, and then uses an emotion engine to recognize the user's emotions, providing household management and related information. Specific means for implementing the present invention and examples of its operation are shown below.

[1374] System Overview

[1375] The system includes a camera, a receiver, a text information extractor, an analyzer, a database registerer, a displayer, a push notification system, and an emotion recognition system. The system is designed to allow users to easily manage their household finances and achieve efficient spending management and savings. Furthermore, the system's usability is enhanced by recognizing the user's emotions and adjusting the system's behavior and notification content accordingly.

[1376] Photograph and send receipt

[1377] User

[1378] The user launches the household accounting mini-app on their smartphone or tablet and takes a picture of the receipt using the device's camera. For example, they can press the "take a picture of receipt" button on the app's home screen to switch to the camera screen. When the user focuses on the receipt and presses the shutter button, the captured image is sent to the server.

[1379] Receipt image reception and analysis

[1380] server

[1381] The server receives the receipt image sent by the user. It then uses OCR technology such as Tesseract to extract text information from the image. For example, a string of characters in the format "Super X, Tomato, ¥200, October 5, 2023" is generated. This text information is then broken down into detailed data for the database using an analysis tool.

[1382] Classification and storage of generated data

[1383] server

[1384] The server classifies the analyzed data into categories such as "grocery" and "daily necessities." For example, data classified as "grocery" is stored in a MySQL database.

[1385] Update and view household accounts

[1386] Terminal

[1387] The device periodically sends a request to the server to retrieve updated information. If new data is available, it is automatically reflected in the on-screen household ledger display, allowing the user to view it in real time.

[1388] Accumulation and analysis of purchase history

[1389] server

[1390] The server collects past purchase data and analyzes it based on specific items (e.g., by store, frequency of product purchases, etc.). The results of this analysis visualize the purchasing trends of each user.

[1391] Providing discount information

[1392] server

[1393] The server compares the household account data with the advertising database to obtain information such as "Products at a specific store are 10% off this week." This information is then sent to the user as a push notification.

[1394] Operation of emotion recognition means

[1395] server

[1396] The server analyzes the user's facial expressions and voice data in real time to recognize emotions. Using a deep learning model, the user's emotional state is classified into categories such as "surprise," "sadness," and "happiness."

[1397] Adjustments to household account display and notifications

[1398] server

[1399] If the server determines that the user is feeling stressed based on the analysis results of the emotion recognition means, it will provide a visually easy-to-read interface and send push notifications with information on benefits for relaxation.

[1400] Specific examples

[1401] For example, when a user goes shopping at a supermarket and receives a receipt, the following process takes place: The user launches the household accounting mini-app and takes a photo of the receipt. The image is sent to the server, and OCR technology extracts the data: "Supermarket X, Tomatoes, 200 yen, October 5, 2023." The analysis means classifies this data into categories (e.g., "Groceries") and stores it in a database. The user can check the latest household accounting information on their device and understand their spending status by category and overall. Furthermore, if the emotion recognition means detects that the user is stressed, an interface is provided that allows them to easily check their spending status for that day, and relevant coupons for the next day are pushed to them.

[1402] Prompt Sentence Examples

[1403] "Please explain the process of a system that analyzes receipt images taken by users and automatically updates the household account book."

[1404] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1405] Step 1:

[1406] Photograph and send receipt

[1407] The user launches the household account book mini-app and clicks the "Take a photo of receipt" button on the home screen, which launches the camera screen.

[1408] The user focuses on the receipt and presses the shutter button to take a photo of the receipt.

[1409] Input: A receipt image captured by the camera.

[1410] After confirming the image, the user presses the "send" button, and the captured image is sent to the server.

[1411] Output: Receipt image data sent to the server.

[1412] Step 2:

[1413] Receipt image reception and analysis

[1414] The server receives the receipt image sent by the user.

[1415] Input: Receipt image data sent from the user device.

[1416] The server uses OCR technology (e.g., Tesseract OCR) to extract text information from the receipt image.

[1417] Data processing: The process of extracting text information from receipt image data.

[1418] The server analyzes the extracted text information and converts it into detailed data such as the store name, product name, price, and purchase date and time.

[1419] Output: Text information such as store name, product name, price, purchase date and time.

[1420] Step 3:

[1421] Classification and storage of generated data

[1422] The server categorizes the analyzed text information into categories, such as "foodstuffs" and "daily necessities."

[1423] Input: Analyzed text information (store name, product name, price, purchase date and time, etc.).

[1424] The server stores this information in a database (e.g., a MySQL database).

[1425] Data processing: The process of classifying data by category and storing it in a database.

[1426] Output: Database storage of character information categorized by category.

[1427] Step 4:

[1428] Update and view household accounts

[1429] The device periodically sends requests to the server to obtain updated household accounting information.

[1430] Input: Updated household accounting information provided by the server.

[1431] If the device has new data, it will automatically be reflected in the household ledger display on the screen.

[1432] Data processing: Display processing of new household accounting data.

[1433] Output: The updated household accounting information is displayed on the user's device.

[1434] Step 5:

[1435] Accumulation and analysis of purchase history

[1436] The server collects the purchase history stored in the household account book database and performs analysis based on specific items (e.g., by store, frequency of purchase of product, etc.).

[1437] Input: Past purchase history stored in the household accounting database.

[1438] The server aggregates this data and visualizes each user's purchasing trends.

[1439] Data processing: Analysis and aggregation of purchase history data.

[1440] Output: Purchasing trend data based on the analysis results.

[1441] Step 6:

[1442] Providing discount information

[1443] The server compares the household account book database with other information databases to obtain discount information relevant to the user.

[1444] Input: Information from the household accounting database and advertising database.

[1445] The server provides relevant information to the user via push notifications.

[1446] Data processing: Cross-database information matching and notification content generation.

[1447] Output: Deals sent via push notification.

[1448] Step 7:

[1449] Operation of emotion recognition means

[1450] The user allows access to the device's camera and microphone.

[1451] The server uses emotion recognition means to analyze the user's facial expressions and voice data and recognize emotions.

[1452] Input: User's facial expression data and voice data.

[1453] The server uses a deep learning model to identify the emotional state.

[1454] Data processing: Analysis and processing of facial expression and voice data.

[1455] Output: Identified emotional state data.

[1456] Step 8:

[1457] Adjustments to household account display and notifications

[1458] The server adjusts the visualization method of the household accounting data and the notification content based on the results of the emotion recognition means.

[1459] Input: Identified emotional state data and household ledger information.

[1460] The server generates an interface and notification content that matches the user's emotions.

[1461] Data processing: Adjustment of display and notification content based on emotional state.

[1462] Output: Adjusted interface and push notification content.

[1463] (Application example 2)

[1464] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1465] In conventional household management systems, entering expenditure data is tedious, and it is particularly difficult to fill out receipts while driving or on the go. Furthermore, the systems do not take the user's feelings into consideration, resulting in poor usability. Furthermore, they do not adequately provide advertisements or discount information linked to receipt information, making it difficult for users to save money efficiently. To solve these problems, it is necessary to improve the user experience in household management systems.

[1466] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1467] In this invention, the server includes a photographing means for a user to photograph a receipt, a receiving means for receiving the receipt image photographed by the photographing means, a character information extraction means for extracting character information from the receipt image using OCR technology, an analysis means for analyzing the character information extracted by the character information extraction means and generating a store name, product name, price, and purchase date and time, a database registration means for categorizing the information generated by the analysis means and storing it in a household accounting database, a display means for displaying the information stored in the household accounting database, a push notification means for comparing the household accounting database with an advertising database and providing relevant discount information via push notification, an emotion recognition means for recognizing the user's emotion, an adjustment means for adjusting the display method of the household accounting data and the notification content based on the emotion analyzed by the emotion recognition means, and an integration means integrated into the infotainment system of an autonomous vehicle and coordinating with an in-vehicle camera and a voice assistant. This enables safe household management while driving and provides optimal information tailored to the user's emotion.

[1468] A "photographing means" is a device used by a user to capture an image of a printed medium such as a receipt.

[1469] The "receiving means" is a function for importing the receipt image captured by the image capturing means into the system.

[1470] The "character information extraction means" is a function that uses OCR technology to recognize and extract characters from a captured receipt image.

[1471] The "analysis means" is a function that analyzes the extracted character information and converts it into specific data such as the store name, product name, price, and purchase date and time.

[1472] The "database registration means" is a function for classifying the information generated by the analysis means into categories and storing the information in the household account book database.

[1473] The "display means" is a device or function for visually presenting the information stored in the household accounting database to the user.

[1474] The "push notification means" is a function that checks the household account book database against the advertisement database and notifies the user of relevant deals.

[1475] The "emotion recognition means" is a function for analyzing and recognizing emotions from the user's voice, facial expressions, etc.

[1476] The "adjustment means" is a function that adjusts the display method of household accounting data and notification content based on the user's emotions analyzed by the emotion recognition means.

[1477] "Integration means" refers to the functionality for integrating and linking other systems and devices with the infotainment system of an autonomous vehicle.

[1478] The present invention is a system that is integrated into the infotainment system of an autonomous vehicle, allowing users to safely and efficiently manage their household finances while driving. The system mainly includes a photographing means, a receiving means, a text information extraction means, an analysis means, a database registration means, a display means, a push notification means, an emotion recognition means, an adjustment means, and an integration means.

[1479] Hardware and Software Used

[1480] Hardware:

[1481] In-car camera

[1482] microphone

[1483] Touch panel display

[1484] In-vehicle computer

[1485] software:

[1486] OCR engine (e.g. Tesseract)

[1487] Emotion recognition engine (e.g. Microsoft Azure Cognitive Services)

[1488] Speech synthesis engine (e.g. Google Text-to-Speech API)

[1489] Cloud databases (e.g. Firebase)

[1490] Infotainment System API

[1491] System Operation

[1492] User

[1493] After shopping, users can take a photo of the receipt with the car's camera, ensuring safety while driving.

[1494] server

[1495] 1. Receipt reception and OCR processing:

[1496] The receiving device captures the receipt image taken by the in-car camera, and the OCR engine extracts text information from the image. This process generates data such as the store name, product name, price, and purchase date and time.

[1497] 2. Data analysis and classification:

[1498] The analysis means analyzes various data using the extracted character information, and the database registration means stores the information in the household account book database. At this time, the data is automatically classified into categories such as food, daily necessities, and beverages.

[1499] 3. Matching purchase history with advertising data:

[1500] The server compares the household account database with the advertisement database to generate relevant deals for the user, which are then provided to the user via push notification.

[1501] 4. Emotion recognition:

[1502] An emotion recognition engine analyzes voice data collected through microphones in the car, thereby identifying the user's current emotion (happiness, surprise, sadness, anger, etc.).

[1503] 5. Notification Adjustments:

[1504] The application adjusts the display method of household accounting data and notification content based on the emotion information acquired by the emotion recognition means, thereby providing optimal information suited to the user.

[1505] Terminal

[1506] The device (touch panel display of the infotainment system) periodically retrieves the latest household accounting information sent from the server and visually displays it. It also has a function to read out the contents of push notifications using a speech synthesis engine.

[1507] Specific examples

[1508] For example, if a user is driving an autonomous vehicle and purchases coffee at a drive-thru and receives a receipt, they simply present the receipt to the in-car camera. The onboard computer uses an OCR engine to extract information such as "coffee shop, coffee, 400 yen, October 10, 2023." The data is then stored and categorized in a Firebase cloud database. The infotainment system's touchscreen display displays the latest household finances and provides relevant deals via voice. If the emotion recognition engine determines the user is stressed, the system will notify them with a gentle voice message encouraging them to relax.

[1509] Prompt Sentence Examples

[1510] When a user takes a photo of a receipt with the in-car camera, the OCR engine extracts the text information and stores it in Firebase. The emotion recognition engine analyzes the user's voice and identifies their emotion. Next, the household accounting information is displayed on the touch panel, and if necessary, the speech synthesis engine provides discount information.

[1511] This system allows users to intuitively and safely manage their household finances while driving, and provides optimal information based on their emotional state. This technology is specialized for autonomous vehicles, contributing to an improved user experience.

[1512] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1513] Step 1:

[1514] The user takes a photo of the receipt

[1515] After shopping, the user presents the receipt to the camera in the autonomous vehicle and takes a photo. The input is the receipt image, and the output is the transfer of image data to the on-board computer. User interaction in this step is kept to a minimum.

[1516] Step 2:

[1517] Receiving receipt images

[1518] The server (on-board computer) receives receipt images taken by the user with the in-car camera. The input is the receipt image, and the output is image data provided to the OCR engine. This step ensures that the receipt image is captured in the system.

[1519] Step 3:

[1520] OCR analysis

[1521] Extract text information from the receipt image using an OCR engine (e.g., Tesseract). The input is the receipt image, and the output is the extracted text information (store name, product name, price, purchase date and time). This step converts the image data into text data.

[1522] Step 4:

[1523] Data analysis

[1524] The server analyzes the extracted text and generates the appropriate information (store name, product name, price, purchase date and time). The input is text, and the output is the analyzed data. This step involves formatting and categorizing the data.

[1525] Step 5:

[1526] Saving to a database

[1527] The database registration means classifies the analyzed information by category and stores it in the household account book database. The input is the analyzed data, and the output is the database registration of the classified data. This allows the user to check their expenses by category.

[1528] Step 6:

[1529] Displaying household accounts

[1530] The terminal (the display of the in-vehicle infotainment system) periodically retrieves the latest household accounting information from the server and displays it. The input is information from the household accounting database, and the output is a visual presentation of the information on the display. At this step, the user can check the expenditure information.

[1531] Step 7:

[1532] Matching purchase history with advertising data

[1533] The server compares the household account database with the advertisement database and generates deals related to the user. The input is household account data and advertisement data, and the output is deals. This step generates information useful to the user.

[1534] Step 8:

[1535] Push notifications

[1536] The push notification means notifies the user of the discount information generated in the previous step. The input is the discount information, and the output is a real-time notification to the user. In this step, the user receives the information efficiently.

[1537] Step 9:

[1538] emotion recognition

[1539] The emotion recognition means analyzes the voice data collected using the in-car microphone and identifies the user's emotion. The input is voice data, and the output is the user's emotional information. This step identifies the user's emotional state.

[1540] Step 10:

[1541] Adjusting notification content

[1542] The server adjusts the display method of the household accounting data and the notification content based on the emotional information analyzed by the emotion recognition means. The input is the user's emotional information, and the output is the adjusted display method and notification content. This step provides the user with the most appropriate information.

[1543] The above is a detailed description of each processing step, which allows the system to provide advanced household management and emotion-based services while ensuring user safety.

[1544] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1545] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1546] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1547] [Fourth embodiment]

[1548] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1549] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1550] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1551] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1552] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1553] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1554] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1555] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1556] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1557] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1558] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1559] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1560] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1561] MODE FOR CARRYING OUT THE INVENTION

[1562] The present invention provides a system that automatically generates a household account book by allowing a user to simply take a photo of a receipt. The following is a specific example for implementing the present invention.

[1563] System Overview

[1564] The purpose of this system is to enable users to manage their household finances hassle-free. The system includes a photographing means, a receiving means, a text information extraction means, an analysis means, a database registration means, a display means, and a push notification means. This allows users to easily grasp their spending situation, obtain future spending forecasts, and obtain advantageous information.

[1565] Explanation of program processing

[1566] Photograph and send receipt

[1567] User

[1568] The user launches the household account book mini-app and takes a picture of the receipt using the camera function, which then sends the image to the server.

[1569] Receipt image reception and analysis

[1570] server

[1571] The server receives the received receipt image. Next, it uses OCR technology to extract text information from the receipt image. The extracted text information is then analyzed and converted into detailed information such as the store name, product name, price, and purchase date and time.

[1572] Classification and storage of generated data

[1573] server

[1574] The server categorizes the analyzed information and stores it in a household accounting database, which is continuously updated to accumulate the user's purchasing history.

[1575] Update and view household accounts

[1576] Terminal

[1577] The terminal receives the latest household accounting information from the server and displays it to the user, allowing the user to easily manage their household finances.

[1578] Accumulation and analysis of purchase history

[1579] server

[1580] The server analyzes the purchase history stored in the household accounting database and generates data such as the purchase frequency of specific stores and products.

[1581] Providing discount information

[1582] server

[1583] The server compares the household accounting database with the advertising database and delivers relevant information via push notifications, which users can use to save money efficiently.

[1584] Specific examples

[1585] For example, if a user goes shopping at a supermarket and receives a receipt, the following happens: The user launches the mini-app and takes a photo of the receipt. The image is sent to the server, and OCR technology extracts data such as "Supermarket X, Tomatoes, 200 yen, October 5, 2023." The analysis means categorizes this data into categories (e.g., "Groceries") and stores it in a database. The user's device displays the data the next day and is also notified that Supermarket X's tomatoes will be on sale over the weekend.

[1586] The above is an embodiment of the present invention, which allows the user to manage their household finances effectively and efficiently.

[1587] The processing flow will be explained below.

[1588] Step 1:

[1589] User

[1590] The user launches the household account book mini-app and selects the camera function. They take a picture of the receipt with the camera and press the capture button. A confirmation screen is then displayed, and the user confirms the image and presses the send button.

[1591] Step 2:

[1592] Terminal

[1593] The device then sends the captured receipt image to the server, where it is encrypted to ensure data security.

[1594] Step 3:

[1595] server

[1596] The server receives the receipt image sent by the user using a receiving means. Once receipt is confirmed, the image is converted into text information using OCR (Optical Character Recognition) technology.

[1597] Step 4:

[1598] server

[1599] The character information extracted by OCR technology is processed by an analysis means to generate detailed information such as the store name, product name, price, purchase date, etc. At this stage, recognition errors are also checked and corrected.

[1600] Step 5:

[1601] server

[1602] The analyzed information is classified by category. For example, it is automatically sorted into categories such as groceries, daily necessities, and beverages. The sorted data is saved in a household accounting database.

[1603] Step 6:

[1604] Terminal

[1605] The device periodically retrieves updated household accounting information from the server, and any new data is automatically reflected in the user's app.

[1606] Step 7:

[1607] User

[1608] Users can check their latest household accounting information within the app and understand their income and expenditure situation. Expenses by category and overall spending status are displayed in graph and list format.

[1609] Step 8:

[1610] server

[1611] The system analyzes a user's purchasing history based on data from the previous month and past data, generating analysis results such as purchase frequency of specific stores and products, and monthly spending trends.

[1612] Step 9:

[1613] server

[1614] The system compares the household accounting database with the advertising database to identify deals relevant to the user. For example, if a frequently purchased item is on sale, that information is extracted.

[1615] Step 10:

[1616] Terminal

[1617] The device receives push notifications from the server and displays them to the user, who can view them within the app and can find specific coupons or sales information.

[1618] Step 11:

[1619] User

[1620] Users can check the notifications and take advantage of the displayed deals, making it easy to save money by using coupons and plan their shopping based on the sale information.

[1621] Example 1

[1622] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1623] Traditional household management methods are time-consuming and require users to manually input data, resulting in problems with accuracy and efficiency. Furthermore, few systems offer added value, such as predicting future spending based on purchase history or providing information on special offers. This makes it difficult for users to effectively manage their household finances and budgets.

[1624] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1625] In this invention, the server includes an image capture means for a user to take a picture of a receipt, an image receiving means for receiving the receipt image captured by the image capture means, a character information extraction means for extracting character information from the receipt image using optical character recognition technology, an information analysis means for analyzing the character information extracted by the character information extraction means and generating a store name, product name, price, and purchase date and time, a database recording means for categorizing the information generated by the information analysis means and storing it in a financial management database, an information display means for displaying the information stored in the financial management database, and a notification means for comparing the financial management database with an advertising database and providing relevant discount information via push notification. This allows users to efficiently manage their household finances simply by taking pictures of receipts, and also enables them to receive future spending forecasts and special offers based on their purchase history.

[1626] "Image capture means" refers to a device or method that allows a user to take a photo of a receipt.

[1627] "Image receiving means" refers to a device or method for receiving a captured receipt image from the user's terminal.

[1628] Optical character recognition (OCR) is a technology that extracts text information from an image.

[1629] "Text information extraction means" refers to a device or method for extracting text information from a receipt image.

[1630] The "information analysis means" refers to a device or method that analyzes the extracted character information and generates detailed information such as the store name, product name, price, and purchase date and time.

[1631] The "database recording means" refers to a device or method for categorizing the analyzed information and storing it in a financial management database.

[1632] "Information display means" refers to a device or method for displaying information stored in the financial management database to a user.

[1633] The "notification means" refers to a device or method that cross-references the financial management database with the advertising database and provides relevant discount information to the user via push notification.

[1634] The present invention is a system that automatically generates a household account book by allowing a user to simply take a photo of a receipt. The following is a specific example for implementing the present invention.

[1635] System Overview

[1636] The purpose of this system is to allow users to manage their household finances hassle-free. The system includes the following components:

[1637] 1. Image acquisition method

[1638] 2. Image Receiving Method

[1639] 3. Optical character recognition technology (OCR)

[1640] 4. Text information extraction method

[1641] 5. Information analysis means

[1642] 6. Database Recording Method

[1643] 7. Information display means

[1644] 8. Means of notification

[1645] Photograph and send receipt

[1646] User

[1647] The user launches the dedicated household accounting mini-app. Next, they take a photo of the receipt using the smartphone's camera. This involves the user operating the camera and adjusting it so that the entire receipt is captured clearly. Once the photo is taken, the app provides instructions for sending the image to the server. The user presses the send button to send the image to the server.

[1648] Receipt image reception and analysis

[1649] server

[1650] The server receives the receipt image sent from the user's device. This reception process includes a method to ensure security using the HTTPS protocol. Next, the server uses the Tesseract OCR library to extract text information from the received image. This OCR process generates text data, which provides detailed information such as the store name, product name, price, and purchase date and time. Furthermore, the extracted text information is structured and organized by category using information analysis means.

[1651] Classification and storage of generated data

[1652] server

[1653] The server classifies the data generated by the information analysis means into categories, such as food, beverages, and daily necessities. This classified data is stored in a financial management database using a database recording means. This database uses a relational database management system such as MySQL or PostgreSQL. The data is continuously updated, and user purchase histories are accumulated.

[1654] Update and view household accounts

[1655] Terminal

[1656] The device periodically accesses the server to obtain the latest household accounting data. Data is obtained via communication via a RESTful API. The received data is temporarily stored in the device's local storage and displayed on a dedicated UI (user interface). This display process uses UI libraries such as React Native. On this screen, the user can view a list of expenses by day and category.

[1657] Accumulation and analysis of purchase history

[1658] server

[1659] The server periodically analyzes the purchase history data stored in the database. This analysis is performed using Python data processing libraries (Pandas and NumPy). For example, graphs are created to show the purchase frequency of specific products and trends in total spending. The results of this analysis are used to generate individual reports for each user.

[1660] Providing discount information

[1661] server

[1662] The server compares the financial management database with the advertising database to generate relevant discount information. This process uses ad distribution services such as the AdSense API. The generated discount information is then provided to users via push notifications. Firebase Cloud Messaging (FCM) is used for push notifications. For example, the server can notify users in real time of sale information for products they frequently purchase.

[1663] Specific examples

[1664] For example, consider the case where a user goes shopping at a supermarket and receives a receipt. The user launches the household accounting mini-app and takes a picture of the receipt using the smartphone's camera. The image is sent to the server, and OCR technology extracts text information such as "Supermarket X, Tomatoes, 200 yen, October 5, 2023." Analysis means categorizes this data into the "grocery" category and stores it in a database. The next day, the user's device receives the latest household accounting data and displays it on the screen. Additionally, a push notification is sent informing them that Supermarket X tomatoes will be on sale over the weekend.

[1665] Example prompts to input to the generative AI model

[1666] Using prompts like the following can effectively leverage generative AI models (e.g., GPT-4):

[1667] "Please explain the automatic household accounting system. The user takes a photo of a receipt and sends the image to a server. The server uses OCR technology to extract text information from the receipt, analyzes it, and saves it in a database. The device receives and displays the latest household accounting data, and the server analyzes purchase history and provides savings information."

[1668] The above is a specific embodiment for carrying out the present invention. This system allows the user to manage their household finances effectively and efficiently.

[1669] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1670] Step 1:

[1671] User

[1672] The user launches the household account book mini-app and takes a photo of the receipt using the smartphone's camera. The user adjusts the camera so that the entire receipt is clearly visible, and then presses the "take a photo" button.

[1673] Input: Receipt image taken by the camera via user operation

[1674] Output: Photographed receipt image data

[1675] Step 2:

[1676] User

[1677] The user presses the "Send" button in the app to send the captured image of the receipt to the server. The app encodes the image data and sends it to the server using the HTTPS protocol.

[1678] Input: Photographed receipt image data

[1679] Output: Receipt image data sent to the server

[1680] Step 3:

[1681] server

[1682] The server receives the sent receipt image. This reception uses the HTTPS protocol to ensure security. The received image data is temporarily stored in server storage.

[1683] Input: Receipt image data sent by the user

[1684] Output: Receipt image data stored on the server

[1685] Step 4:

[1686] server

[1687] The server calls the Tesseract OCR library to extract text information from the received receipt image. The OCR process generates the receipt's text data.

[1688] Input: Receipt image data stored on the server

[1689] Output: Extracted text information (store name, product name, price, purchase date and time, etc.)

[1690] Step 5:

[1691] server

[1692] The server analyzes the text information to identify the store name, product name, price, purchase date, etc. It also categorizes the information into categories, such as food, beverages, and daily necessities.

[1693] Input: Extracted text information

[1694] Output: Parsed and categorized information

[1695] Step 6:

[1696] server

[1697] The server stores the classified information in a financial management database, which also checks for duplicate data. The database uses a relational database management system (MySQL or PostgreSQL).

[1698] Input: Parsed and classified information

[1699] Output: Purchase history information stored in the database

[1700] Step 7:

[1701] Terminal

[1702] The device periodically accesses the server to retrieve the latest household accounting data. This process uses communication via a RESTful API. The retrieved data is temporarily stored in local storage and displayed in a dedicated UI.

[1703] Input: Latest household accounting data obtained from the server

[1704] Output: Household accounting information displayed on the terminal

[1705] Step 8:

[1706] server

[1707] The server periodically analyzes the purchase history data stored in the financial management database. This analysis uses Python data processing libraries (Pandas and NumPy) to generate data such as the purchase frequency of specific products and trends in total spending.

[1708] Input: Purchase history data stored in the financial management database

[1709] Output: Analysis results (purchase frequency, trends in total expenditure, etc.)

[1710] Step 9:

[1711] server

[1712] The server compares the financial management database with the advertising database to generate relevant discount information. This is done using an ad distribution service such as the AdSense API. The generated discount information is then sent to users via push notifications.

[1713] Input: Analysis results, advertising database information

[1714] Output: A push notification with the discount information sent to the user.

[1715] This allows users to efficiently manage their household finances simply by taking photos of receipts, and they can also receive future spending forecasts and special offers based on their purchasing history.

[1716] (Application example 1)

[1717] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1718] In recent years, with the spread of e-commerce, an increasing number of users use multiple online shopping sites. However, manually managing the electronic receipts received from each online shopping site is cumbersome, resulting in a decrease in the efficiency of users' household finance management. In addition, it is difficult to integrate and manage online and offline purchase data, making it difficult to efficiently forecast expenses and manage budgets. This increases the burden on users when managing their household finances.

[1719] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1720] In this invention, the server includes a photographing device for a user to photograph a receipt, a receiving device for receiving the receipt image photographed by the photographing device, a character information extraction device for extracting character information from the receipt image using OCR technology, an analysis device for analyzing the character information extracted by the character information extraction device and generating a store name, product name, price, and purchase date and time, a database registration device for categorizing the information generated by the analysis device and storing it in a household accounting database, a display device for displaying the information stored in the household accounting database, a push notification device for comparing the household accounting database with an advertising database and providing relevant discount information via push notification, and a data acquisition device for automatically acquiring and analyzing electronic receipts from online stores. This allows users to automatically acquire, analyze, and manage not only physical receipts but also electronic receipt information from online stores. This not only enables more efficient household management but also improves users' overall spending forecasting and budget management.

[1721] "Photographing means" refers to the device or function that a user uses to photograph a physical receipt.

[1722] The "receiving means" refers to a device or function for transmitting the receipt image captured by the capturing means to the server and receiving it.

[1723] "Text information extraction means" refers to functions or software for extracting text information from receipt images using OCR technology.

[1724] "Analysis means" refers to a function or software that analyzes the extracted character information and generates detailed information such as the store name, product name, price, and purchase date and time.

[1725] The "database registration means" refers to a function or software that classifies the information generated by the analysis means into categories and stores it in the household accounting database.

[1726] "Display means" refers to devices or software that visually present the information stored in the household accounting database to the user.

[1727] The "push notification means" is a function that compares the household accounting database with the advertisement database and provides relevant discount information to the user as a push notification.

[1728] "Data acquisition means" refers to functions or software for automatically acquiring and analyzing electronic receipts from online stores.

[1729] The present invention provides a system that automatically generates a household account book by allowing users to simply take a photo of a receipt. In particular, the present invention has the ability to automatically acquire and analyze electronic receipts from online stores. The following are specific embodiments for carrying out the present invention.

[1730] System Overview

[1731] The purpose of this system is to enable users to manage their household finances hassle-free. The system includes a photographing means, a receiving means, a text information extraction means, an analysis means, a database registration means, a display means, a push notification means, and a data acquisition means.

[1732] Program processing explanation

[1733] Photograph and send receipt

[1734] The user launches the household accounting app and takes a photo of the physical receipt using the smartphone's camera, which then sends the image to the server.

[1735] Receipt image reception and analysis

[1736] The server receives the receipt image and extracts the text information using an OCR library (Tesseract OCR). The extracted text information is then analyzed and converted into detailed information such as the store name, product name, price, and purchase date and time.

[1737] Classification and storage of generated data

[1738] The server categorizes the analyzed information and stores it in a SQLite database, which is continually updated to accumulate the user's purchasing history.

[1739] Update and view household accounts

[1740] The device receives the latest household accounting information from the server and displays it to the user using React Native, allowing users to easily manage their household finances.

[1741] Automatic capture and analysis of electronic receipts

[1742] The server accesses the user's shopping site account and automatically retrieves the electronic receipt. This process can be set to occur periodically. The retrieved electronic receipt is analyzed using OCR technology to extract text information similar to that of a physical receipt.

[1743] Accumulation and analysis of purchase history

[1744] The server analyzes the purchase history stored in the household account book database and generates data such as the frequency of purchases of specific stores and products, which allows the server to understand users' purchasing patterns and trends.

[1745] Providing discount information

[1746] The server compares the household accounting database with the advertising database and delivers relevant deals via push notifications, such as notifications about sales on specific products.

[1747] Specific examples

[1748] For example, if a user purchases an item from an online store and receives an electronic receipt, the following happens: The server automatically checks the purchase history of the online shopping site periodically and retrieves the electronic receipt, even if the user does not launch the household accounting app. OCR technology extracts information such as "online store, product name, 5000 yen, October 5, 2023," and an analysis method classifies this information into a category (for example, "entertainment") and stores it in a database. When the user opens the app, they can check the latest household accounting information and receive notifications when specific products go on sale.

[1749] Prompt Sentence Examples

[1750] "We will develop an application that analyzes digital receipts received from online stores and reflects them in a household ledger. The process involves obtaining purchase history, extracting text information using OCR technology, storing it in a database, and displaying it in a user interface. The tools used are React Native, Tesseract OCR, and SQLite."

[1751] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1752] Step 1:

[1753] The user launches the household accounting app on their smartphone and takes a photo of the physical receipt. By doing this, the user captures the receipt image using the camera function.

[1754] Input: Physical receipt

[1755] Output: Receipt image

[1756] Step 2:

[1757] The server receives the receipt image sent from the smartphone, and the receipt image is stored on the server as a data packet.

[1758] Input: Receipt image

[1759] Output: Receipt image stored on the server

[1760] Step 3:

[1761] The server uses an OCR library (Tesseract OCR) to extract text information from the received receipt image. During this process, the image data is converted into text data.

[1762] Input: Receipt image

[1763] Output: Extracted text information (store name, product name, price, purchase date and time)

[1764] Step 4:

[1765] The server uses an analysis tool to analyze the extracted text information and generate detailed information such as the store name, product name, price, and purchase date and time, allowing each item to be identified and organized as usable data.

[1766] Input: Extracted text information

[1767] Output: Organized detailed information (store name, product name, price, purchase date and time)

[1768] Step 5:

[1769] The server categorizes the details into categories (e.g., groceries, entertainment) and stores them in an SQLite database. During this process, the data is classified and registered for storage.

[1770] Input: Organized details

[1771] Output: Database records containing information categorized by category

[1772] Step 6:

[1773] The server accesses the user's online shopping site account and automatically retrieves electronic receipts. It periodically checks the purchase history and downloads new electronic receipts.

[1774] Input: User's online shopping site account information

[1775] Output: The obtained e-receipt

[1776] Step 7:

[1777] The server analyzes the acquired electronic receipt using OCR technology to extract text information. Just like physical receipts, text data is generated from electronic receipts.

[1778] Input: Retrieved e-receipt

[1779] Output: Extracted text information (store name, product name, price, purchase date and time)

[1780] Step 8:

[1781] The server categorizes the text information extracted from the electronic receipts and stores it in a database, where it is managed together with the physical receipt data.

[1782] Input: Extracted text information (electronic receipt)

[1783] Output: Database records containing categorized e-receipt information

[1784] Step 9:

[1785] The device receives the latest household accounting data from the server and displays it to the user. React Native is used to provide an intuitive visual interface.

[1786] Input: Latest household accounting data received from the server

[1787] Output: User interface reflecting household accounting data

[1788] Step 10:

[1789] The server compares the household accounting database with the advertising database and pushes relevant deals to users, providing them with useful information in real time.

[1790] Input: household accounting database, advertising database

[1791] Output: Push notification of deals

[1792] By following these steps, users can automatically manage physical receipts and electronic receipts from online stores, and efficiently update and use their household accounts.

[1793] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1794] MODE FOR CARRYING OUT THE INVENTION

[1795] The present invention is a system that automatically generates a household account book when a user simply takes a photo of a receipt, and then uses an emotion engine to recognize the user's emotions, manages household finances, and provides information on special offers. The following is a specific example of how the present invention can be implemented.

[1796] System Overview

[1797] The system includes a camera, a receiver, a text information extraction unit, an analysis unit, a database registration unit, a display unit, a push notification unit, and an emotion engine. The system is designed to allow users to easily manage their household finances and efficiently manage and save money. Furthermore, the system recognizes the user's emotions and adjusts the system's behavior and notification content accordingly, improving ease of use.

[1798] Explanation of program processing

[1799] Photograph and send receipt

[1800] User

[1801] The user launches the household account book mini-app and takes a picture of the receipt using the camera function, which then sends the image to the server.

[1802] Receipt image reception and analysis

[1803] server

[1804] The server receives the received receipt image. Next, it uses OCR technology to extract text information from the receipt image. The extracted text information is then analyzed and converted into detailed information such as the store name, product name, price, and purchase date and time.

[1805] Classification and storage of generated data

[1806] server

[1807] The server categorizes the analyzed information. For example, it automatically sorts items into categories such as groceries, daily necessities, and beverages. The sorted data is then saved in a household accounting database.

[1808] Update and view household accounts

[1809] Terminal

[1810] The device periodically retrieves updated household accounting information from the server, and any new data is automatically reflected in the user's app.

[1811] Accumulation and analysis of purchase history

[1812] server

[1813] The server analyzes the purchase history stored in the household accounting database and generates data such as the purchase frequency of specific stores and products.

[1814] Providing discount information

[1815] server

[1816] The server compares the household accounting database with the advertising database and provides relevant discount information to users via push notifications, which can be used to help users save money efficiently.

[1817] Emotion Engine Operation

[1818] server

[1819] The server runs an emotion engine based on the user's facial and voice data to recognize the user's emotions. For example, it uses a camera and microphone to analyze facial expressions and tone of voice to identify emotions such as joy, surprise, sadness, and anger.

[1820] Adjustments to household account display and notifications

[1821] server

[1822] The server adjusts the visualization of the household accounting data and the content of push notifications based on the user's emotions recognized by the emotion engine. For example, if the user is feeling stressed, it will notify them with an encouraging message or a simple task.

[1823] Specific examples

[1824] For example, when a user goes shopping at a supermarket and receives a receipt, the following sequence of events takes place: The user launches the mini-app and takes a photo of the receipt. The image is sent to the server, where OCR technology extracts data such as "Supermarket X, Tomatoes, 200 yen, October 5, 2023." The analysis tool categorizes this data into categories (e.g., "Groceries") and stores it in a database. The user can check the latest household accounting information displayed on the device and understand spending by category and overall spending. Additionally, if the emotion engine detects stress in the user, an interface is displayed that allows them to easily check their spending for the day, and relevant coupons for the next day are sent via push notification.

[1825] The above is an embodiment of the present invention, which allows users to effectively and efficiently manage their finances and receive support tailored to their emotional state.

[1826] The processing flow will be explained below.

[1827] Step 1:

[1828] User

[1829] The user launches the household account book mini-app and selects the camera function. They take a picture of the receipt with the camera and press the capture button. A confirmation screen is then displayed, and the user confirms the image and presses the send button.

[1830] Step 2:

[1831] Terminal

[1832] The device then sends the captured receipt image to the server, where it is encrypted to ensure data security.

[1833] Step 3:

[1834] server

[1835] The server receives the receipt image sent by the user using a receiving means. Once receipt is confirmed, the image is converted into text information using OCR (Optical Character Recognition) technology.

[1836] Step 4:

[1837] server

[1838] The character information extracted by OCR technology is processed by an analysis means to generate detailed information such as the store name, product name, price, purchase date, etc. At this stage, recognition errors are also checked and corrected.

[1839] Step 5:

[1840] server

[1841] The analyzed information is classified by category. For example, it is automatically sorted into categories such as groceries, daily necessities, and beverages. The sorted data is saved in a household accounting database.

[1842] Step 6:

[1843] Terminal

[1844] The device periodically retrieves updated household accounting information from the server, and any new data is automatically reflected in the user's app.

[1845] Step 7:

[1846] User

[1847] Users can check their latest household accounting information within the app and understand their income and expenditure situation. Expenses by category and overall spending status are displayed in graph and list format.

[1848] Step 8:

[1849] server

[1850] The server runs an emotion engine based on the user's facial and voice data to recognize the user's emotions. For example, it uses a camera and microphone to analyze facial expressions and tone of voice to identify emotions such as joy, surprise, sadness, and anger.

[1851] Step 9:

[1852] server

[1853] The emotion engine adjusts the visualization of household accounting data based on the user's emotions. If the user is feeling stressed, the data display will be simplified and the colors will be softened.

[1854] Step 10:

[1855] server

[1856] The system compares the household accounting database with the advertising database to identify deals relevant to the user. For example, if a frequently purchased item is on sale, that information is extracted.

[1857] Step 11:

[1858] Terminal

[1859] The device receives push notifications from the server and displays them to the user, who can view them within the app and can find specific coupons or sales information.

[1860] Step 12:

[1861] User

[1862] Users can check the notifications and take advantage of the displayed deals, making it easy to save money by using coupons and plan their shopping based on the sale information.

[1863] Step 13:

[1864] server

[1865] The emotion engine periodically analyzes the user's emotions and adjusts the content and timing of push notifications. For example, if the user is feeling stressed, it will provide encouraging messages or information to help them relax.

[1866] The above is a specific embodiment of the present invention that combines an emotion engine, allowing users to effectively manage their finances and receive support according to their emotional state.

[1867] Example 2

[1868] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1869] Modern society demands tools that allow individuals to efficiently and effectively manage their household finances. It is particularly important to easily record and classify daily shopping expenses and use them to predict future spending. Furthermore, providing support that takes into account the user's emotional state would improve user satisfaction and ease of use. However, current household finance management systems rarely meet all of these requirements, and they often place a heavy burden on users. Therefore, the challenge is to provide a system that makes it easy for users to manage their spending and provides the necessary information while taking their emotions into account.

[1870] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1871] In this invention, the server includes: a photographing means for a user to photograph a receipt; a receiving means for receiving the receipt image photographed by the photographing means; a character information extraction means for extracting character information from the receipt image using OCR technology; an analysis means for analyzing the character information extracted by the character information extraction means and generating a store name, product name, price, and purchase date and time; a database registration means for categorizing the information generated by the analysis means and storing it in a database; a display means for displaying the information stored in the database; a push notification means for comparing the database with other information databases and providing highly relevant information via push notification; an emotion recognition means for analyzing the user's facial expressions and voice data to recognize emotions; and an adjustment means for adjusting the display content of the display means and the notification content of the push notification means based on the emotion recognized by the emotion recognition means. This not only allows users to easily record and manage their expenses, but also provides them with optimal information according to their emotional state, making household management more efficient and effective.

[1872] "Photographing means" refers to a function or device that allows a user to photograph a receipt with a camera.

[1873] The "receiving means" refers to a function or device for sending image data of a photographed receipt to a server and receiving it.

[1874] "Text information extraction means" refers to the functions and techniques for extracting text information from receipt images using OCR technology.

[1875] The "analysis means" refers to a function or technology for analyzing the character information acquired by the character information extraction means and converting it into detailed data such as the store name, product name, price, and purchase date and time.

[1876] The "database registration means" refers to the function or technology for classifying the information generated by the analysis means into categories and storing it in a database.

[1877] "Display means" refers to functions and technologies for visually presenting information stored in a database to a user.

[1878] "Push notification means" refers to a function or technology that compares the household accounting database with other information databases and provides relevant information to the user via push notification.

[1879] "Emotion recognition means" refers to functions and technologies for analyzing a user's facial expressions and voice data and recognizing the user's emotions.

[1880] The "adjustment means" refers to a function or technology for adjusting the display content of the display means or the notification content of the push notification means based on the emotion recognized by the emotion recognition means.

[1881] MODE FOR CARRYING OUT THE INVENTION

[1882] The present invention is a system that automatically generates a household account book simply by a user taking a photo of a receipt, and then uses an emotion engine to recognize the user's emotions, providing household management and related information. Specific means for implementing the present invention and examples of its operation are shown below.

[1883] System Overview

[1884] The system includes a camera, a receiver, a text information extractor, an analyzer, a database registerer, a displayer, a push notification system, and an emotion recognition system. The system is designed to allow users to easily manage their household finances and achieve efficient spending management and savings. Furthermore, the system's usability is enhanced by recognizing the user's emotions and adjusting the system's behavior and notification content accordingly.

[1885] Photograph and send receipt

[1886] User

[1887] The user launches the household accounting mini-app on their smartphone or tablet and takes a picture of the receipt using the device's camera. For example, they can press the "take a picture of receipt" button on the app's home screen to switch to the camera screen. When the user focuses on the receipt and presses the shutter button, the captured image is sent to the server.

[1888] Receipt image reception and analysis

[1889] server

[1890] The server receives the receipt image sent by the user. It then uses OCR technology such as Tesseract to extract text information from the image. For example, a string of characters in the format "Super X, Tomato, ¥200, October 5, 2023" is generated. This text information is then broken down into detailed data for the database using an analysis tool.

[1891] Classification and storage of generated data

[1892] server

[1893] The server classifies the analyzed data into categories such as "grocery" and "daily necessities." For example, data classified as "grocery" is stored in a MySQL database.

[1894] Update and view household accounts

[1895] Terminal

[1896] The device periodically sends a request to the server to retrieve updated information. If new data is available, it is automatically reflected in the on-screen household ledger display, allowing the user to view it in real time.

[1897] Accumulation and analysis of purchase history

[1898] server

[1899] The server collects past purchase data and analyzes it based on specific items (e.g., by store, frequency of product purchases, etc.). The results of this analysis visualize the purchasing trends of each user.

[1900] Providing discount information

[1901] server

[1902] The server compares the household account data with the advertising database to obtain information such as "Products at a specific store are 10% off this week." This information is then sent to the user as a push notification.

[1903] Operation of emotion recognition means

[1904] server

[1905] The server analyzes the user's facial expressions and voice data in real time to recognize emotions. Using a deep learning model, the user's emotional state is classified into categories such as "surprise," "sadness," and "happiness."

[1906] Adjustments to household account display and notifications

[1907] server

[1908] If the server determines that the user is feeling stressed based on the analysis results of the emotion recognition means, it will provide a visually easy-to-read interface and send push notifications with information on benefits for relaxation.

[1909] Specific examples

[1910] For example, when a user goes shopping at a supermarket and receives a receipt, the following process takes place: The user launches the household accounting mini-app and takes a photo of the receipt. The image is sent to the server, and OCR technology extracts the data: "Supermarket X, Tomatoes, 200 yen, October 5, 2023." The analysis means classifies this data into categories (e.g., "Groceries") and stores it in a database. The user can check the latest household accounting information on their device and understand their spending status by category and overall. Furthermore, if the emotion recognition means detects that the user is stressed, an interface is provided that allows them to easily check their spending status for that day, and relevant coupons for the next day are pushed to them.

[1911] Prompt Sentence Examples

[1912] "Please explain the process of a system that analyzes receipt images taken by users and automatically updates the household account book."

[1913] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1914] Step 1:

[1915] Photograph and send receipt

[1916] The user launches the household account book mini-app and clicks the "Take a photo of receipt" button on the home screen, which launches the camera screen.

[1917] The user focuses on the receipt and presses the shutter button to take a photo of the receipt.

[1918] Input: A receipt image captured by the camera.

[1919] After confirming the image, the user presses the "send" button, and the captured image is sent to the server.

[1920] Output: Receipt image data sent to the server.

[1921] Step 2:

[1922] Receipt image reception and analysis

[1923] The server receives the receipt image sent by the user.

[1924] Input: Receipt image data sent from the user device.

[1925] The server uses OCR technology (e.g., Tesseract OCR) to extract text information from the receipt image.

[1926] Data processing: The process of extracting text information from receipt image data.

[1927] The server analyzes the extracted text information and converts it into detailed data such as the store name, product name, price, and purchase date and time.

[1928] Output: Text information such as store name, product name, price, purchase date and time.

[1929] Step 3:

[1930] Classification and storage of generated data

[1931] The server categorizes the analyzed text information into categories, such as "foodstuffs" and "daily necessities."

[1932] Input: Analyzed text information (store name, product name, price, purchase date and time, etc.).

[1933] The server stores this information in a database (e.g., a MySQL database).

[1934] Data processing: The process of classifying data by category and storing it in a database.

[1935] Output: Database storage of character information categorized by category.

[1936] Step 4:

[1937] Update and view household accounts

[1938] The device periodically sends requests to the server to obtain updated household accounting information.

[1939] Input: Updated household accounting information provided by the server.

[1940] If the device has new data, it will automatically be reflected in the household ledger display on the screen.

[1941] Data processing: Display processing of new household accounting data.

[1942] Output: The updated household accounting information is displayed on the user's device.

[1943] Step 5:

[1944] Accumulation and analysis of purchase history

[1945] The server collects the purchase history stored in the household account book database and performs analysis based on specific items (e.g., by store, frequency of purchase of product, etc.).

[1946] Input: Past purchase history stored in the household accounting database.

[1947] The server aggregates this data and visualizes each user's purchasing trends.

[1948] Data processing: Analysis and aggregation of purchase history data.

[1949] Output: Purchasing trend data based on the analysis results.

[1950] Step 6:

[1951] Providing discount information

[1952] The server compares the household account book database with other information databases to obtain discount information relevant to the user.

[1953] Input: Information from the household accounting database and advertising database.

[1954] The server provides relevant information to the user via push notifications.

[1955] Data processing: Cross-database information matching and notification content generation.

[1956] Output: Deals sent via push notification.

[1957] Step 7:

[1958] Operation of emotion recognition means

[1959] The user allows access to the device's camera and microphone.

[1960] The server uses emotion recognition means to analyze the user's facial expressions and voice data and recognize emotions.

[1961] Input: User's facial expression data and voice data.

[1962] The server uses a deep learning model to identify the emotional state.

[1963] Data processing: Analysis and processing of facial expression and voice data.

[1964] Output: Identified emotional state data.

[1965] Step 8:

[1966] Adjustments to household account display and notifications

[1967] The server adjusts the visualization method of the household accounting data and the notification content based on the results of the emotion recognition means.

[1968] Input: Identified emotional state data and household ledger information.

[1969] The server generates an interface and notification content that matches the user's emotions.

[1970] Data processing: Adjustment of display and notification content based on emotional state.

[1971] Output: Adjusted interface and push notification content.

[1972] (Application example 2)

[1973] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1974] In conventional household management systems, entering expenditure data is tedious, and it is particularly difficult to fill out receipts while driving or on the go. Furthermore, the systems do not take the user's feelings into consideration, resulting in poor usability. Furthermore, they do not adequately provide advertisements or discount information linked to receipt information, making it difficult for users to save money efficiently. To solve these problems, it is necessary to improve the user experience in household management systems.

[1975] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1976] In this invention, the server includes a photographing means for a user to photograph a receipt, a receiving means for receiving the receipt image photographed by the photographing means, a character information extraction means for extracting character information from the receipt image using OCR technology, an analysis means for analyzing the character information extracted by the character information extraction means and generating a store name, product name, price, and purchase date and time, a database registration means for categorizing the information generated by the analysis means and storing it in a household accounting database, a display means for displaying the information stored in the household accounting database, a push notification means for comparing the household accounting database with an advertising database and providing relevant discount information via push notification, an emotion recognition means for recognizing the user's emotion, an adjustment means for adjusting the display method of the household accounting data and the notification content based on the emotion analyzed by the emotion recognition means, and an integration means integrated into the infotainment system of an autonomous vehicle and coordinating with an in-vehicle camera and a voice assistant. This enables safe household management while driving and provides optimal information tailored to the user's emotion.

[1977] A "photographing means" is a device used by a user to capture an image of a printed medium such as a receipt.

[1978] The "receiving means" is a function for importing the receipt image captured by the image capturing means into the system.

[1979] The "character information extraction means" is a function that uses OCR technology to recognize and extract characters from a captured receipt image.

[1980] The "analysis means" is a function that analyzes the extracted character information and converts it into specific data such as the store name, product name, price, and purchase date and time.

[1981] The "database registration means" is a function for classifying the information generated by the analysis means into categories and storing the information in the household account book database.

[1982] The "display means" is a device or function for visually presenting the information stored in the household accounting database to the user.

[1983] The "push notification means" is a function that checks the household account book database against the advertisement database and notifies the user of relevant deals.

[1984] The "emotion recognition means" is a function for analyzing and recognizing emotions from the user's voice, facial expressions, etc.

[1985] The "adjustment means" is a function that adjusts the display method of household accounting data and notification content based on the user's emotions analyzed by the emotion recognition means.

[1986] "Integration means" refers to the functionality for integrating and linking other systems and devices with the infotainment system of an autonomous vehicle.

[1987] The present invention is a system that is integrated into the infotainment system of an autonomous vehicle, allowing users to safely and efficiently manage their household finances while driving. The system mainly includes a photographing means, a receiving means, a text information extraction means, an analysis means, a database registration means, a display means, a push notification means, an emotion recognition means, an adjustment means, and an integration means.

[1988] Hardware and Software Used

[1989] Hardware:

[1990] In-car camera

[1991] microphone

[1992] Touch panel display

[1993] In-vehicle computer

[1994] software:

[1995] OCR engine (e.g. Tesseract)

[1996] Emotion recognition engine (e.g. Microsoft Azure Cognitive Services)

[1997] Speech synthesis engine (e.g. Google Text-to-Speech API)

[1998] Cloud databases (e.g. Firebase)

[1999] Infotainment System API

[2000] System Operation

[2001] User

[2002] After shopping, users can take a photo of the receipt with the car's camera, ensuring safety while driving.

[2003] server

[2004] 1. Receipt reception and OCR processing:

[2005] The receiving device captures the receipt image taken by the in-car camera, and the OCR engine extracts text information from the image. This process generates data such as the store name, product name, price, and purchase date and time.

[2006] 2. Data analysis and classification:

[2007] The analysis means analyzes various data using the extracted character information, and the database registration means stores the information in the household account book database. At this time, the data is automatically classified into categories such as food, daily necessities, and beverages.

[2008] 3. Matching purchase history with advertising data:

[2009] The server compares the household account database with the advertisement database to generate relevant deals for the user, which are then provided to the user via push notification.

[2010] 4. Emotion recognition:

[2011] An emotion recognition engine analyzes voice data collected through microphones in the car, thereby identifying the user's current emotion (happiness, surprise, sadness, anger, etc.).

[2012] 5. Notification Adjustments:

[2013] The application adjusts the display method of household accounting data and notification content based on the emotion information acquired by the emotion recognition means, thereby providing optimal information suited to the user.

[2014] Terminal

[2015] The device (touch panel display of the infotainment system) periodically retrieves the latest household accounting information sent from the server and visually displays it. It also has a function to read out the contents of push notifications using a speech synthesis engine.

[2016] Specific examples

[2017] For example, if a user is driving an autonomous vehicle and purchases coffee at a drive-thru and receives a receipt, they simply present the receipt to the in-car camera. The onboard computer uses an OCR engine to extract information such as "coffee shop, coffee, 400 yen, October 10, 2023." The data is then stored and categorized in a Firebase cloud database. The infotainment system's touchscreen display displays the latest household finances and provides relevant deals via voice. If the emotion recognition engine determines the user is stressed, the system will notify them with a gentle voice message encouraging them to relax.

[2018] Prompt Sentence Examples

[2019] When a user takes a photo of a receipt with the in-car camera, the OCR engine extracts the text information and stores it in Firebase. The emotion recognition engine analyzes the user's voice and identifies their emotion. Next, the household accounting information is displayed on the touch panel, and if necessary, the speech synthesis engine provides discount information.

[2020] This system allows users to intuitively and safely manage their household finances while driving, and provides optimal information based on their emotional state. This technology is specialized for autonomous vehicles, contributing to an improved user experience.

[2021] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2022] Step 1:

[2023] The user takes a photo of the receipt

[2024] After shopping, the user presents the receipt to the camera in the autonomous vehicle and takes a photo. The input is the receipt image, and the output is the transfer of image data to the on-board computer. User interaction in this step is kept to a minimum.

[2025] Step 2:

[2026] Receiving receipt images

[2027] The server (on-board computer) receives receipt images taken by the user with the in-car camera. The input is the receipt image, and the output is image data provided to the OCR engine. This step ensures that the receipt image is captured in the system.

[2028] Step 3:

[2029] OCR analysis

[2030] Extract text information from the receipt image using an OCR engine (e.g., Tesseract). The input is the receipt image, and the output is the extracted text information (store name, product name, price, purchase date and time). This step converts the image data into text data.

[2031] Step 4:

[2032] Data analysis

[2033] The server analyzes the extracted text and generates the appropriate information (store name, product name, price, purchase date and time). The input is text, and the output is the analyzed data. This step involves formatting and categorizing the data.

[2034] Step 5:

[2035] Saving to a database

[2036] The database registration means classifies the analyzed information by category and stores it in the household account book database. The input is the analyzed data, and the output is the database registration of the classified data. This allows the user to check their expenses by category.

[2037] Step 6:

[2038] Displaying household accounts

[2039] The terminal (the display of the in-vehicle infotainment system) periodically retrieves the latest household accounting information from the server and displays it. The input is information from the household accounting database, and the output is a visual presentation of the information on the display. At this step, the user can check the expenditure information.

[2040] Step 7:

[2041] Matching purchase history with advertising data

[2042] The server compares the household account database with the advertisement database and generates deals related to the user. The input is household account data and advertisement data, and the output is deals. This step generates information useful to the user.

[2043] Step 8:

[2044] Push notifications

[2045] The push notification means notifies the user of the discount information generated in the previous step. The input is the discount information, and the output is a real-time notification to the user. In this step, the user receives the information efficiently.

[2046] Step 9:

[2047] emotion recognition

[2048] The emotion recognition means analyzes the voice data collected using the in-car microphone and identifies the user's emotion. The input is voice data, and the output is the user's emotional information. This step identifies the user's emotional state.

[2049] Step 10:

[2050] Adjusting notification content

[2051] The server adjusts the display method of the household accounting data and the notification content based on the emotional information analyzed by the emotion recognition means. The input is the user's emotional information, and the output is the adjusted display method and notification content. This step provides the user with the most appropriate information.

[2052] The above is a detailed description of each processing step, which allows the system to provide advanced household management and emotion-based services while ensuring user safety.

[2053] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2054] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2055] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2056] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2057] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2058] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2059] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2060] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2061] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2062] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2063] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2064] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2065] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. Fo...

Claims

1. A photographing means for a user to photograph a receipt; a receiving means for receiving the receipt image captured by the imaging means; A character information extraction means for extracting character information from the receipt image using OCR technology; an analysis means for analyzing the character information extracted by the character information extraction means and generating a store name, a product name, a price, and a purchase date and time; a database registration means for classifying the information generated by the analysis means into categories and storing the information in a household account book database; a display means for displaying information stored in the household account book database; a push notification means for comparing the household account book database with an advertisement database and providing highly relevant advantageous information by push notification; A system including:

2. 2. The system according to claim 1, further comprising an expenditure forecasting means for forecasting future expenditures based on the information generated by said analysis means, thereby supporting budget management.

3. 2. The system according to claim 1, wherein said household account book database includes a history analysis means for storing a user's purchase history and analyzing the purchase frequency of a particular store or product.

Images