system

Abstract

Provide a system. A method for capturing a paper document as a digital image; character recognition means for converting captured image data into character information; a data classification means for classifying character information and converting it into structured data; database means for storing the data converted by the data classification means; a search input means for a user to input a search query; search result display means for searching a database based on the search query and displaying the results; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern homes, a large number of paper documents are often stored in a disorganized manner, making it difficult to quickly find the information you need. Managing a wide range of documents, including utility bills, shopping receipts, medical statements, insurance documents, school notices, and pension newsletters, can be particularly cumbersome and hinders efficient living. Furthermore, it is often impossible to access important information immediately, which can be time-consuming and stressful. This can lead to a decline in quality of life, overlooking important information, and wasting time, making it an urgent task to address this issue. [Means for solving the problem]

[0005] This invention provides a system that includes a means for capturing paper documents as digital images, a character recognition means for converting the captured image data into text information, a data classification means for classifying the text information and converting it into structured data, a database means for storing the data converted by the data classification means, a search input means for a user to input a search query, and a search result display means for searching the database based on the search query and displaying the results. The system also includes an information recommendation means for recommending information based on frequency of use and importance, and a summary generation means for generating summaries of the search results. This system efficiently digitizes a wide variety of paper documents found in the home, facilitating rapid access to that information and supporting the understanding and utilization of important information. This system significantly simplifies paper document management, reducing the time burden and stress on users and enabling them to live an efficient and smart life.

[0006] "Paper document" means a document or record that is printed or handwritten on paper.

[0007] "Digital image" refers to image data that has been converted from analog information into a digital format.

[0008] "Character recognition means" refers to techniques and algorithms used to extract textual information from digital images.

[0009] "Data classification means" refers to the technology or algorithm used to classify extracted text information and organize it as structured data.

[0010] "Database means" refers to systems or software for efficiently storing, managing, and retrieving structured data.

[0011] A "search input method" is an interface or device through which a user enters a search query.

[0012] "Search result display means" refers to an interface or device for displaying to a user the results obtained based on a search query.

[0013] "Information recommendation means" refers to technologies and algorithms for selecting information based on frequency of use and importance and recommending it to users.

[0014] "Summary generation means" refers to the technology or algorithm used to create a summary based on search results. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] As an embodiment of the present invention, a system for digitizing paper documents present in the home and managing, searching, summarizing, reporting, and recommending information will be described. This system has the following configuration and functions.

[0037] System configuration

[0038] 1. Paper document capture method

[0039] Users use a smartphone or scanner to take a photo of a paper document (utility bill, shopping receipt, medical bill, insurance document, school notice, pension newsletter, etc.) and capture it as a digital image.

[0040] 2. Character recognition method (OCR)

[0041] The server receives the digital image sent by the user and uses OCR technology to extract text information from the image. Character recognition is performed after preprocessing such as noise removal and skew correction.

[0042] 3. Data Classification Methods

[0043] The server analyzes the extracted text and classifies the data based on the document type (utility bill, receipt, medical statement, etc.) using machine learning algorithms.

[0044] 4. Database Means

[0045] The server stores the classified data in a database. The stored data is structured and managed in the form of pairs of item names and values.

[0046] 5. Search Input Methods

[0047] A user enters a search query into the application's search bar, for example, "electricity bill for the last 3 months."

[0048] 6. Search result display method

[0049] The server searches the database based on the search query and displays the relevant data to the user, optionally generating and displaying a summary of the search results.

[0050] 7. Information recommendation means

[0051] The server analyzes the information based on the frequency and importance of the user's use and recommends appropriate information to the user, such as notifying them that their insurance contract is due for renewal.

[0052] System operation example

[0053] 1. Utility bill management

[0054] Users take a photo of their electricity bill with their smartphone and upload it to a dedicated app.

[0055] The device sends the uploaded image to the server.

[0056] The server uses OCR technology to extract text from the image and classify it into the "utility bills" category.

[0057] The server stores the data in a database so that it can be quickly accessed when a user searches.

[0058] 2. Search for medical details

[0059] If a user wants to check the details of medical expenses for the past year, they enter "medical expenses for the past year" into the application.

[0060] The terminal sends a search query to the server, which searches the database for the relevant medical details.

[0061] The server displays the search results and their summaries to the user.

[0062] 3. Create monthly reports from receipts

[0063] If a user wants to know the total cost of food for the month, he / she inputs the request "Total cost of food for this month" into the application.

[0064] The terminal sends a request to the server, and the server aggregates the data for the specified period.

[0065] The server generates a report of the aggregated results and provides it to the user.

[0066] 4. Recommendation of insurance documents

[0067] The server analyzes the user's past search history and identifies important insurance-related information.

[0068] The server notifies the user that the time for renewal of the insurance contract is approaching and recommends related information to the user.

[0069] The user checks the recommended information in the app and takes the necessary action.

[0070] Summary

[0071] This invention allows users to efficiently digitize paper documents and quickly access the information contained therein. In particular, it allows users to manage their lives smartly and efficiently through search, summarization, reporting, and information recommendation functions. This system is expected to improve the quality of life by eliminating the complexity of managing paper documents for users and reducing time and stress.

[0072] The processing flow will be explained below.

[0073] Step 1:

[0074] Users take a photo of a paper document using a smartphone or scanner and import it as a digital image using a dedicated application.

[0075] Step 2:

[0076] The device sends the captured image to the server in an optimized format, compressing the image and adjusting the resolution to optimize the transfer speed.

[0077] Step 3:

[0078] The server temporarily stores the received digital images, along with the image's file format and metadata.

[0079] Step 4:

[0080] The server extracts text information from images using OCR (Optical Character Recognition) technology. Specifically, after preprocessing the image (noise removal, tilt correction, binarization, etc.), it detects text areas and recognizes the characters from them.

[0081] Step 5:

[0082] The server analyzes the extracted text and classifies the data based on the type of document, such as into categories like "utility bill," "receipt," or "medical statement," using machine learning algorithms.

[0083] Step 6:

[0084] The server organizes the classified data as structured data and stores it in a database. The data is stored in fields such as date, amount, and item name.

[0085] Step 7:

[0086] A user enters a search query through an application, for example, "electricity bill details for the last 3 months."

[0087] Step 8:

[0088] The terminal sends the user's search query to the server.

[0089] Step 9:

[0090] The server searches the database for data corresponding to the search query, using indexes to efficiently extract the data.

[0091] Step 10:

[0092] The server creates a summary of the search results using a summary generation algorithm as needed.

[0093] Step 11:

[0094] The server transmits the generated search results and summaries to the terminal.

[0095] Step 12:

[0096] The terminal displays search results and summaries to the user, allowing the user to quickly access the information they need.

[0097] Step 13:

[0098] The server analyzes the user's past search history, frequency of use, access patterns, etc. Based on this, it applies an algorithm to recommend important information and actions.

[0099] Step 14:

[0100] The server recommends information that it deems important to the user, and this information is sent to the device as a notification.

[0101] Step 15:

[0102] The terminal displays a notification of the recommended information to the user and provides an interface that allows the user to view detailed information.

[0103] These are the processing steps for a system that digitizes paper documents and performs management, search, summarization, reporting, and information recommendation, thereby simplifying paper document management for users and enabling quick access to the information they need.

[0104] Example 1

[0105] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0106] In today's world, homes are overflowing with paper documents, making it difficult to manage them efficiently. Manually searching through documents to find the information you need takes time and effort. It's also difficult to identify useful information based on its importance and frequency of use. There's a need for a system that can solve these issues and enable users to digitize paper documents and efficiently manage, search, summarize, report, and recommend information.

[0107] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0108] In this invention, the server includes a means for capturing paper documents as digital images, a means for converting the captured image data into text information, a means for classifying the text information and converting it into structured data, a data storage means, a means for a user to input a search query, a means for searching a data recording device based on the search query and displaying the results, a means for summarizing data based on the user's search query, and a means for recommending information based on user behavior data. This allows users to efficiently digitize paper documents and easily access the information. Furthermore, the system enables users to manage their lives smartly and efficiently through search, summarization, reporting, and information recommendations.

[0109] "Paper documents" refers to printed or written forms of paper, including electricity bills, shopping receipts, medical statements, insurance documents, school notices, pension newsletters, etc.

[0110] A "digital image" refers to an image file generated by photographing or scanning a physical paper document with a device such as a smartphone or scanner.

[0111] "Capture means" refers to a device and application software for capturing a paper document as a digital image.

[0112] "Character recognition means" refers to a technology that analyzes character information in an image and converts it into character data, and generally uses optical character recognition (OCR) technology.

[0113] "Data classification means" refers to algorithms and techniques for classifying extracted textual information into different categories (e.g., utility bills, receipts, medical statements, etc.).

[0114] "Data storage means" refers to a database system for long-term storage of information after data classification, which appropriately structures and stores digital data.

[0115] "Search input means" refers to an interface through which a user inputs a query to search for specific information, and sends this to a server via a terminal.

[0116] "Search result display means" refers to a function for displaying to a user data retrieved based on a search query, optionally providing results in a summarized form.

[0117] "Summary generation means" refers to the technology and algorithms used to summarize search results and information contained in a database and present it to the user in an easy-to-understand manner.

[0118] "Information recommendation means" refers to technology for recommending appropriate information and actions based on a user's usage history and behavioral data.

[0119] The system of the present invention is designed to efficiently digitize paper documents in the home and perform management, retrieval, summarization, reporting, and information recommendation. The structure of this system and its technical implementation are described in detail below.

[0120] System configuration and usage

[0121] Paper document capture method

[0122] Users use digital devices such as smartphones or scanners to capture digital images of paper documents (e.g., utility bills, receipts, medical statements, insurance documents, etc.) and can upload these digital images to the system through a dedicated application.

[0123] Character recognition means (OCR)

[0124] The digital image uploaded by the device is sent to a server, which then uses OCR technology, such as Amazon Textract or Google Cloud Vision, to extract text information from the image. The OCR process includes preprocessing steps such as noise removal and skew correction.

[0125] Data Classification Methods

[0126] The server analyzes the extracted text and classifies the data based on the document type, using machine learning algorithms such as support vector machines (SVM) and random forests, for example, into categories such as medical statements, receipts, and utility bills.

[0127] Data storage means

[0128] The server stores the classified data in a database. The data is structured as pairs of item names and values. The database uses a database management system such as MySQL (registered trademark) or PostgreSQL.

[0129] Search input method

[0130] The user enters a search query into the application's search bar. For example, the user might enter a query such as "electricity bill details for the past three months." The device then sends this search query to the server.

[0131] Search result display method

[0132] The server searches the database based on the received search query and extracts the relevant data. The search results are summarized and displayed to the user. The search function uses technologies such as Lucene and ElasticSearch (registered trademark).

[0133] Information recommendation means

[0134] The server analyzes the information based on the user's usage history and importance, and recommends appropriate information. For example, it could provide a function to notify users when their insurance contract is due for renewal.

[0135] Specific operation example

[0136] Utility bill management

[0137] Example prompt: "Please upload a photo of your electricity bill taken with your smartphone."

[0138] Users take a photo of their electricity bill with their smartphone and upload it to a dedicated app.

[0139] The device sends the uploaded image to the server.

[0140] The server uses OCR technology to extract text information from the image and classify it into the "utility bills" category.

[0141] The server stores the data in a database.

[0142] Search for medical details

[0143] Example prompt: "Please search for medical expense details for the past year."

[0144] A user types "medical expenses past year" into the app's search bar.

[0145] The device sends a search query to the server.

[0146] The server searches the database, extracts and summarizes the relevant medical details, and displays them to the user.

[0147] Generate monthly reports from receipts

[0148] Example prompt: "What is the total cost of food this month?"

[0149] The user enters "Total food expenses this month" into the app.

[0150] The device sends a query to the server.

[0151] The server aggregates the data for the specified period and generates a report.

[0152] The server sends the report to the user.

[0153] Recommendation of insurance documents

[0154] Example prompt: "I would like to receive insurance policy renewal notices."

[0155] The server analyzes the user's past search history and identifies important insurance-related information.

[0156] The server notifies the user that the time for renewal of the insurance contract is approaching and recommends related information to the user.

[0157] The user checks the recommended information in the app and takes the necessary action.

[0158] The system allows users to efficiently digitize paper documents and access information more quickly and easily, while search result summaries and information recommendations help users manage their lives more efficiently.

[0159] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0160] Step 1:

[0161] A user captures a paper document as a digital image using a smartphone or scanner.

[0162] Input: Paper document

[0163] Output: Digital image

[0164] Specific actions: The user launches the smartphone's camera app and takes a photo of a document, or scans a document using a scanner. The user then imports the captured or scanned image into the app.

[0165] Step 2:

[0166] The device uploads the digital images taken or scanned to a server via a dedicated app.

[0167] Input: Digital image

[0168] Output: Digital image sent to server

[0169] Specific operation: The user clicks the "Upload" button on the dedicated app to send the photographed or scanned image data to the cloud server. The device displays the sending status to the user.

[0170] Step 3:

[0171] The server performs OCR processing on the received digital image to extract text information.

[0172] Input: Digital image

[0173] Output: Extracted text information

[0174] How it works: The server analyzes the received image data using OCR technology such as Amazon Textract or Google Cloud Vision. The OCR process includes noise removal and tilt correction, and the character information in the image is extracted as text data.

[0175] Step 4:

[0176] The server analyzes the extracted text information and categorizes it based on the document type.

[0177] Input: Extracted text information

[0178] Output: Categorized data

[0179] How it works: The server analyzes text information using machine learning algorithms such as SVM and Random Forest. For example, the server identifies the keyword "electricity bill" and classifies it into the "utility bill" category.

[0180] Step 5:

[0181] The server stores the classified data in a database.

[0182] Input: Categorical data

[0183] Output: Data stored in the database

[0184] How it works: The server uses a database management system such as MySQL or PostgreSQL to store data categorized into categories in the form of item name / value pairs. For example, data in the "Utility Expenses" category is recorded in the form of "Date," "Amount," "Category," etc.

[0185] Step 6:

[0186] A user enters a search query into the search bar of an application, and the terminal sends the search query to a server.

[0187] Input: search query

[0188] Output: The search query sent to the server

[0189] What happens: The user enters "electricity bill details for the last 3 months" into the application's search bar and clicks the search button. The device sends the search query to the server.

[0190] Step 7:

[0191] The server searches the database based on the received search query, extracts and summarizes the relevant data.

[0192] Input: search query

[0193] Output: Summarized search results

[0194] What it does: The server uses technologies such as Lucene or Elasticsearch to search the database, extracts relevant data, and optionally summarizes the search results using an automatic summarization algorithm.

[0195] Step 8:

[0196] The server sends the summarized search results to the terminal, which displays the results to the user.

[0197] Input: Summarized search results

[0198] Output: The result displayed to the user

[0199] Specific operation: The server sends search results to the device in JSON format, etc. The device displays the received results in an easy-to-read format within the application interface, and the user confirms the search results.

[0200] Step 9:

[0201] The server analyzes the information based on the frequency of use and importance of the user, and recommends appropriate information to the user.

[0202] Input: User behavior data and usage history

[0203] Output: Recommendation notification

[0204] What happens: The server uses machine learning algorithms to analyze user behavior data. For example, it identifies when an insurance contract is due for renewal and notifies the user. The user then checks the recommendations in the app and takes the necessary action.

[0205] (Application example 1)

[0206] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0207] In recent years, the number of users of online shopping sites has increased, increasing the need for efficient management of purchase history and order details. However, managing paper-based statements is time-consuming, and it is not easy to search, summarize, or recommend information. In particular, it is difficult to summarize past purchase history or accurately recommend the next purchase date or related products. For this reason, there is a need for a system that improves user convenience and enables efficient data management and information provision.

[0208] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0209] In this invention, the server includes means for capturing paper documents as digital images, character recognition means for converting the captured image data into character information, data classification means for classifying the character information and converting it into structured data, database means for saving the data converted by the data classification means, search input means for a user to input a search query, search result display means for searching the database based on the search query and displaying the results, and information recommendation means for processing the user's purchase history and recommending the next purchase date and related products. This allows the user to efficiently manage their purchase history and order details and quickly receive searches, summaries, and recommendations.

[0210] "Means for capturing paper documents as digital images" refers to a function that allows a user to photograph a paper document using a smartphone or scanner and capture the image as a digital image.

[0211] "Character recognition means for converting captured image data into character information" is a function that uses OCR technology to extract character information from digital images.

[0212] The "data classification means for classifying character information and converting it into structured data" is a function that analyzes extracted character information, classifies the data based on the type and content of the document, and converts it into structured data in the form of item name and value pairs.

[0213] The "database means for storing data converted by the data classification means" is a function for storing structured data in a database so that it can be accessed efficiently later.

[0214] The "search input means by which a user inputs a search query" refers to a function by which a user inputs a search query to search for specific information in a search bar of an application, for example.

[0215] The "search result display means for searching a database based on a search query and displaying the results" is a function for retrieving data corresponding to the search query from a database and displaying the data to the user.

[0216] The "information recommendation means for processing the user's purchase history and recommending the next purchase time and related products" is a function that analyzes the user's past purchase history and recommends the next appropriate purchase time and related products.

[0217] As an embodiment of the present invention, we will explain a system that efficiently manages purchase histories and order details on an online shopping site and performs searches, summarization, and recommendations. This system has the following configuration and functions.

[0218] System configuration

[0219] 1. Paper document capture method

[0220] Users use their smartphone camera or a dedicated scanner to take a picture of the purchase receipt from the online shopping site and import it as a digital image.

[0221] 2. Character recognition method (OCR)

[0222] The server receives the digital image sent by the user and extracts text information from the image using OCR technology (such as pytesseract), as well as preprocessing the image by grayscale conversion and noise reduction.

[0223] 3. Data Classification Methods

[0224] The server analyzes the extracted text and classifies the data based on the document type (food, home appliances, clothing, etc.) and content using machine learning algorithms.

[0225] 4. Database Means

[0226] The server stores the classified data in a database. The stored data is structured and managed in the form of pairs of item names and values.

[0227] 5. Search Input Methods

[0228] A user types a search query into the search bar of a smartphone application, for example, "amount spent on food in the last 3 months."

[0229] 6. Search result display method

[0230] The server searches the database based on the search query and displays the relevant data to the user, optionally generating and displaying a summary of the search results.

[0231] 7. Information recommendation means

[0232] The server processes the user's purchase history and recommends the next purchase time and related products, for example, notifying the user that it is almost time to buy detergent again.

[0233] System operation example

[0234] 1. Managing purchase details

[0235] Users take a photo of their purchase details with their smartphone and upload it to a dedicated app. The device sends the image to a server, which converts it into text using OCR technology. The server then categorizes the information and organizes it into categories such as "food" or "home appliances."

[0236] 2. Search for food purchase amounts

[0237] If a user wants to find out how much they spent on food over the past three months, they simply enter "food spending over the past three months" into the application. The device sends the search query to the server, which then searches the database for the relevant data, summarizes it, and displays it.

[0238] 3. Recommendation for next purchase

[0239] Users receive recommendations for their next purchase and related products based on their purchase history. For example, if detergent stocks are running low, the server will send a notification recommending the next purchase.

[0240] Prompt Sentence Examples

[0241] Example 1: Want to know how much you spent on food over the past three months?

[0242] Input: "Food purchases for the past 3 months"

[0243] Output: "I've spent ¥5000 on food in the past three months."

[0244] Example 2: Recommending your next detergent purchase

[0245] Input: "Next recommended time to purchase detergent"

[0246] Output: "Your detergent stock is running low, so we recommend you buy more next time."

[0247] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0248] Step 1:

[0249] The user takes a photo of the purchase receipt with their smartphone and uploads the image to a dedicated app.

[0250] Input: Image of purchase details taken with a smartphone

[0251] Output: Digital image file

[0252] Specific operation: Use the smartphone camera to take a photo of a paper purchase invoice and upload the image to the server from the application.

[0253] Step 2:

[0254] The server applies OCR technology to convert the received image data into text information.

[0255] Input: Digital images uploaded by users

[0256] Output: Extracted text information

[0257] Specific operation: The server uses an image processing library (e.g., OpenCV) to convert the image to grayscale and perform noise reduction, then performs OCR using pytesseract.

[0258] Step 3:

[0259] The server analyzes the character information extracted by OCR, classifies the data, and structures it.

[0260] Input: Character information extracted by OCR

[0261] Output: Classified structured data

[0262] Specific operation: The server uses a machine learning algorithm to analyze the text information, classify the data by category or item, such as "food" or "home appliances," and structure it in the form of item name / value pairs.

[0263] Step 4:

[0264] The server stores the structured data in a database.

[0265] Input: Categorized and structured data

[0266] Output: Data stored in the database

[0267] Specific operation: The server connects to a database management system (e.g., MySQL) and stores the classified data by item.

[0268] Step 5:

[0269] A user types a search query into the search bar of a smartphone app.

[0270] Input: A search query such as "food spending in the last 3 months"

[0271] Output: None (because the query is sent to the server)

[0272] What happens: A user enters a search query into the application's search bar, and the query is sent to the server.

[0273] Step 6:

[0274] The server searches the database based on the search query and retrieves the relevant data.

[0275] Input: Search query

[0276] Output: Search results

[0277] What happens: The server runs an SQL query against the database to retrieve the relevant data, for example, the food purchase history for the past three months.

[0278] Step 7:

[0279] The server summarizes the retrieved data and displays it to the user.

[0280] Input: Search results

[0281] Output: Search results displayed as a summary

[0282] Specific operation: The server aggregates and analyzes the acquired data and organizes it as a summary. For example, it calculates the total amount of food purchases for the past three months and displays it to the user.

[0283] Step 8:

[0284] The server analyzes the user's purchasing history and recommends the next purchase time and related products.

[0285] Input: User purchase history data

[0286] Output: Recommendation

[0287] How it works: The server uses a recommendation algorithm to analyze the user's purchase history and generate recommendations for the next appropriate purchase time and related products. For example, if detergent stock is low, a notification will be sent recommending the next purchase.

[0288] Prompt Sentence Examples

[0289] Example 1: Want to know how much you spent on food over the past three months?

[0290] Input: "Food purchases for the past 3 months"

[0291] Output: "I've spent ¥5000 on food in the past three months."

[0292] Example 2: Recommending your next detergent purchase

[0293] Input: "Next recommended time to purchase detergent"

[0294] Output: "Your detergent stock is running low, so we recommend you buy more next time."

[0295] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0296] As an embodiment of the present invention, we will explain a system that digitizes paper documents present in the home and performs management, search, summarization, reporting, and information recommendation, as well as a system that combines an emotion engine that recognizes user emotions. This system has the following configuration and functions.

[0297] System configuration

[0298] 1. Paper document capture method

[0299] Users use a smartphone or scanner to take a photo of a paper document (utility bill, shopping receipt, medical bill, insurance document, school notice, pension newsletter, etc.) and capture it as a digital image.

[0300] 2. Character recognition method (OCR)

[0301] The server receives the digital image sent by the user and uses OCR technology to extract text information from the image. Character recognition is performed after preprocessing such as noise removal and skew correction.

[0302] 3. Data Classification Methods

[0303] The server analyzes the extracted text and classifies the data based on the document type (utility bill, receipt, medical statement, etc.) using machine learning algorithms.

[0304] 4. Database Means

[0305] The server stores the classified data in a database. The stored data is structured and managed in the form of pairs of item names and values.

[0306] 5. Search Input Methods

[0307] A user enters a search query into the application's search bar, for example, "electricity bill for the last 3 months."

[0308] 6. Search result display method

[0309] The server searches the database based on the search query and displays the relevant data to the user, optionally generating and displaying a summary of the search results.

[0310] 7. Information recommendation means

[0311] The server analyzes the information based on the frequency and importance of the user's use and recommends appropriate information to the user, such as notifying them that their insurance contract is due for renewal.

[0312] 8. Emotion recognition means

[0313] The server uses technology to recognize the user's emotions, for example, by analyzing the user's emotional state from their input speed, context, facial recognition, etc.

[0314] 9. Emotion Engine

[0315] The server then recommends information and adjusts the summary of search results based on the user's recognized emotions. The emotion engine changes its behavior depending on the user's emotional state, for example, displaying simple but important information when the user is stressed, or providing detailed information when the user is relaxed.

[0316] System operation example

[0317] 1. Utility bill management

[0318] Users take a photo of their electricity bill with their smartphone and upload it to a dedicated app.

[0319] The device sends the uploaded image to the server.

[0320] The server uses OCR technology to extract text from the image and classify it into the "utility bills" category.

[0321] The server stores the data in a database so that it can be quickly accessed when a user searches.

[0322] 2. Search for medical details

[0323] If a user wants to check the details of medical expenses for the past year, they enter "medical expenses for the past year" into the application.

[0324] The terminal sends a search query to the server, which searches the database for the relevant medical details.

[0325] The server displays the search results and their summaries to the user.

[0326] 3. Create monthly reports from receipts

[0327] If a user wants to know the total cost of food for the month, he / she inputs the request "Total cost of food for this month" into the application.

[0328] The terminal sends a request to the server, and the server aggregates the data for the specified period.

[0329] The server generates a report of the aggregated results and provides it to the user.

[0330] 4. Recommendation of insurance documents

[0331] The server analyzes the user's past search history and identifies important insurance-related information.

[0332] The server notifies the user that the time for renewal of the insurance contract is approaching and recommends related information to the user.

[0333] The user checks the recommended information in the app and takes the necessary action.

[0334] 5. Emotion-based information presentation

[0335] The server recognizes the user's emotions, and if the user is feeling stressed, for example, important and concise information is displayed preferentially.

[0336] If you're relaxed, it will show you more information and additional options.

[0337] This allows for optimal information provision according to the user's emotional state.

[0338] Summary

[0339] This invention allows users to efficiently digitize paper documents and quickly access the information they hold. In particular, by incorporating search, summarization, reporting, and information recommendation functions, as well as taking into account the user's emotions, it is possible to provide more personalized information. This system is expected to improve users' quality of life by eliminating the hassle of managing paper documents and reducing the time and stress involved.

[0340] The processing flow will be explained below.

[0341] Step 1:

[0342] Users take a photo of a paper document using a smartphone or scanner and import it as a digital image using a dedicated application.

[0343] Step 2:

[0344] The device sends the captured image to the server in an optimized format, compressing the image and adjusting the resolution to optimize the transfer speed.

[0345] Step 3:

[0346] The server temporarily stores the received digital images, along with the image's file format and metadata.

[0347] Step 4:

[0348] The server extracts text information from images using OCR (Optical Character Recognition) technology. Specifically, after preprocessing the image (noise removal, tilt correction, binarization, etc.), it detects text areas and recognizes the characters from them.

[0349] Step 5:

[0350] The server analyzes the extracted text and classifies the data based on the type of document, such as into categories like "utility bill," "receipt," or "medical statement," using machine learning algorithms.

[0351] Step 6:

[0352] The server organizes the classified data as structured data and stores it in a database. The data is stored in fields such as date, amount, and item name.

[0353] Step 7:

[0354] A user enters a search query through an application, for example, "electricity bill details for the last 3 months."

[0355] Step 8:

[0356] The terminal sends the user's search query to the server.

[0357] Step 9:

[0358] The server searches the database for data corresponding to the search query, using indexes to efficiently extract the data.

[0359] Step 10:

[0360] The server creates a summary of the search results using a summary generation algorithm as needed.

[0361] Step 11:

[0362] The server transmits the generated search results and summaries to the terminal.

[0363] Step 12:

[0364] The terminal displays search results and summaries to the user, allowing the user to quickly access the information they need.

[0365] Step 13:

[0366] The server analyzes the user's past search history, frequency of use, access patterns, etc. Based on this, it applies an algorithm to recommend important information and actions.

[0367] Step 14:

[0368] The server recommends information that it deems important to the user, and this information is sent to the device as a notification.

[0369] Step 15:

[0370] The terminal displays a notification of the recommended information to the user and provides an interface that allows the user to view detailed information.

[0371] Step 16:

[0372] The server uses technology to recognize the user's emotions, for example, by analyzing the user's emotional state from their typing speed, context, facial recognition, etc.

[0373] Step 17:

[0374] The server recommends information and adjusts the summary of search results based on the user's recognized emotions. The emotion engine displays simple, essential information when the user is stressed, and provides detailed information when the user is relaxed.

[0375] Step 18:

[0376] The device displays the most appropriate information according to the user's emotional state, thereby realizing personalized information provision that matches the user's emotional state.

[0377] These are the processing steps of the system that combines the emotion engine. This system simplifies the management of paper documents for users, enables quick access to necessary information, and provides information based on emotions.

[0378] Example 2

[0379] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0380] In conventional paper document management systems, digitizing and searching documents is cumbersome, and the accuracy of summarization and information recommendations is low.In addition, information presentation does not take into account the user's emotional state, and the user experience is not optimized.

[0381] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0382] In this invention, the server includes means for capturing paper documents as digital images, character recognition means for converting the captured image data into character information, data classification means for classifying the character information and converting it into structured data, database means for saving the data converted by the data classification means, search input means for a user to input a search query, search result display means for searching the database based on the search query and displaying the results, and emotion recognition means for recognizing the user's emotion and adjusting the way information is presented. This allows users to efficiently digitize documents, easily search and acquire information, and receive information optimally suited to their emotional state.

[0383] "Means for capturing paper documents as digital images" refers to a device or software that allows a user to photograph a paper document using a smartphone or scanner and capture the image as a digital image.

[0384] "Character recognition means" refers to a technology for extracting character information from a digital image, specifically using OCR (optical character recognition) technology.

[0385] A "data classification means" is an algorithm or machine learning model that analyzes extracted text information and classifies it into specific categories.

[0386] "Database means" refers to a database system for structuring and storing classified data, including SQL and NoSQL databases.

[0387] A "search input means" is an application or system that provides an interface for a user to enter a search query in order to search for specific information.

[0388] The "search result display means" is an interface for displaying to the user the data acquired based on the search query, and displays it as text or graphs on the screen.

[0389] "Emotion recognition means" is a technology for analyzing the user's emotional state, and recognizes emotions using data such as input speed, context, and facial recognition.

[0390] MODE FOR CARRYING OUT THE INVENTION

[0391] The system of the present invention efficiently digitizes paper documents kept at home by users and manages, searches, summarizes, reports, and recommends information. It also includes an emotion engine that recognizes the user's emotional state and provides information accordingly. This system operates using the following hardware and software:

[0392] Hardware and Software Configuration

[0393] 1. Paper document capture method

[0394] Users use a smartphone or scanner to capture digital images of paper documents (such as electricity bills, shopping receipts, medical bills, etc.), and then upload these images to the device using a dedicated application.

[0395] 2. Character recognition means (OCR technology)

[0396] The server receives the digital image sent by the user and uses OCR (Optical Character Recognition) technology to extract text information from the image. Specifically, a library such as Tesseract OCR is used to perform preprocessing such as noise removal and tilt correction before recognizing the text.

[0397] 3. Data Classification Methods

[0398] The server analyzes the text extracted by OCR and determines the document type using machine learning algorithms such as SVM (support vector machine) and neural networks. For example, an electricity bill would be classified as a "utility bill" category.

[0399] 4. Database Means

[0400] The server stores the classified data in a database using a structured database such as MySQL or PostgreSQL, where the data is stored in the form of field name / value pairs, allowing for fast and efficient data searches and report generation.

[0401] 5. Search Input Methods

[0402] The user enters a search query into the application's search bar, for example, a specific query such as "electricity bill details for the last three months." This search query is sent to the server via the device.

[0403] 6. Search result display method

[0404] The server searches the database based on the received search query and extracts the relevant data. It then generates a summary from the extracted data and displays the results on the user's device. For example, a search result for "electricity bill details for the past three months" would display the details for each month and the total amount.

[0405] 7. Information recommendation means

[0406] The server recommends appropriate information based on the user's usage history and importance. For example, if an insurance contract is due for renewal, the server notifies the user of this information and recommends detailed information. This ensures that the user does not miss important information.

[0407] 8. Emotion recognition means

[0408] The server analyzes the user's emotional state based on their input speed, context, facial recognition, etc. As a specific example, it can accurately recognize the user's emotional state by using Python's OpenCV library and various cloud APIs (e.g., Azure's Emotion API). If the input speed is slow or if words indicating stress are detected in the context, it is determined that the user is feeling stressed.

[0409] 9. Emotion Engine

[0410] The server adjusts the way information is presented based on the user's recognized emotions. For example, if the user is stressed, only simple and important information is displayed, while if the user is relaxed, detailed information and additional options are provided. This allows the server to provide optimal information according to the user's emotional state.

[0411] Examples of concrete examples and prompts

[0412] Example 1: Utility bill management

[0413] Users take a photo of their electricity bill with their smartphone and upload it to a dedicated app.

[0414] The device sends the uploaded image to the server.

[0415] The server uses OCR technology to extract text from the image and classify it into the "utility bills" category.

[0416] The server stores the data in a database so that it can be quickly accessed when a user searches.

[0417] Example 2: Searching for medical details

[0418] If a user wants to check the details of medical expenses for the past year, they enter the search query "medical expenses for the past year."

[0419] The terminal sends a search query to the server, which searches the database for the relevant medical details.

[0420] The server displays the search results and their summaries to the user.

[0421] Example 3: Creating a monthly report from receipts

[0422] If the user wants to know the total food expenses for the month, he / she inputs a request such as "Total food expenses for this month."

[0423] The terminal sends a request to the server, and the server aggregates the data for the specified period.

[0424] The server generates a report of the aggregated results and provides it to the user.

[0425] Example 4: Recommending insurance documents

[0426] The server analyzes the user's past search history and identifies important insurance-related information.

[0427] The server notifies the user that the time for renewal of the insurance contract is approaching and recommends related information to the user.

[0428] The user reviews the recommended information and takes the necessary action.

[0429] Example 5: Emotion-based information provision

[0430] The server recognizes the user's emotions, and if the user is feeling stressed, for example, it prioritizes displaying concise and important information.

[0431] If the user is relaxed, provide more information or additional options, which provides optimal information to the user.

[0432] Example prompts to input to the generative AI model

[0433] Please tell me your electricity bill details for the past three months.

[0434] "Calculate the total cost of food last month."

[0435] Please show me your medical expenses for the past year.

[0436] "Can you tell me when my insurance policy is due for renewal?"

[0437] "Show me recommendations based on my current mood."

[0438] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0439] Step 1:

[0440] The user takes a photo of the document and captures it as a digital image.

[0441] A user takes a photo of a paper document using a smartphone or scanner. For example, they take a photo of an electricity bill. Then, they use a dedicated application to upload the captured image file to their device. The input is a paper document, and the output is a digital image file.

[0442] Step 2:

[0443] The device sends the digital image to the server

[0444] The terminal transfers the digital image file uploaded by the user to the server. The input is the digital image file, and the output is the digital image file stored on the server. This completes the communication until the image reaches the server.

[0445] Step 3:

[0446] The server extracts text information using OCR technology.

[0447] The server analyzes the received digital image and extracts text information from the image using OCR technology. Specifically, it uses the Tesseract OCR library and performs preprocessing such as noise removal and tilt correction. The input is the digital image file, and the output is the extracted text information.

[0448] Step 4:

[0449] The server classifies the data

[0450] The server analyzes the text extracted by OCR and uses machine learning algorithms (such as SVM or neural networks) to determine the type of document and classify it. For example, an electricity bill is classified into the "utility bill" category. The input is text, and the output is classified data.

[0451] Step 5:

[0452] The server stores the classified data in a database.

[0453] The server stores the classified data in a database. Specifically, it uses a MySQL or PostgreSQL database and stores the data in a structured format of field name / value pairs. The input is the classified data, and the output is the data stored in the database.

[0454] Step 6:

[0455] The user enters a search query

[0456] A user enters a query into the application's search bar to search for specific information. For example, a specific search query such as "electricity bill details for the last three months." The input is the user's search query, and the output is the transmission of the search query from the device to the server.

[0457] Step 7:

[0458] The server searches the database and displays the results

[0459] The server searches the database based on the received search query and extracts the relevant data. A summary is generated from the extracted data and the results are displayed on the user's device. For example, a search result for "electricity bill details for the past three months" would display the details for each month and the total amount. The input is the search query, and the output is the display of the search results.

[0460] Step 8:

[0461] Server recommends information

[0462] The server recommends appropriate information based on the user's usage history and importance. For example, if an insurance contract is due for renewal, the server notifies the user of this information and recommends detailed information. This prevents the user from missing important information. The input is the user's usage history and importance, and the output is a notification of recommended information.

[0463] Step 9:

[0464] The server recognizes emotions and adjusts information

[0465] The server analyzes and recognizes the user's emotional state based on their input speed, context, and facial recognition. Specifically, it uses the Python OpenCV library and Azure's Emotion API. For example, if the input speed is slow or if the user uses many words indicating stress, it determines that the user is feeling stressed. The input is the user's input data and video data, and the output is the analysis result of the user's emotional state.

[0466] Step 10:

[0467] The server provides information using an emotion engine

[0468] The server adjusts the way information is presented based on the user's recognized emotions. For example, if the user is stressed, it displays only simple and important information, while if the user is relaxed, it provides detailed information and additional options. The input is the analysis result of the user's emotional state, and the output is the adjusted information provided.

[0469] (Application example 2)

[0470] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0471] In conventional paper document management systems, managing paper coupons and receipts is cumbersome, making it difficult to digitize them for easy user use. Furthermore, providing information according to the user's emotional state is not considered, which does not lead to an improved user experience. This has led to a demand for a system that allows users to efficiently manage paper information and quickly obtain the information they need without feeling stressed.

[0472] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0473] In this invention, the server includes means for capturing paper documents as digital images, character recognition means for converting the captured image data into character information, data classification means for classifying the character information and converting it into structured data, database means for saving the data converted by the data classification means, search input means for a user to input a search query, search result display means for searching the database based on the search query and displaying the results, emotion recognition means for recognizing the emotional state of the user, and emotion engine means for adjusting the method of providing information based on the recognized emotional state. This enables users to efficiently digitize and manage paper coupons and receipts, and further enables providing information according to the user's emotional state, thereby providing a more comfortable and personalized user experience.

[0474] - "Means for capturing paper documents" means means for capturing paper documents as digital images.

[0475] The "character recognition means" is a means for converting captured digital image data into character information.

[0476] The "data classification means" is a means for classifying the character information converted by the character recognition means into categories and converting it into structured data.

[0477] "Database Means" means a means for storing classified structured data and making it accessible as needed.

[0478] A "search input means" is a means by which a user inputs a search query.

[0479] The "search result display means" is a means for searching a database based on an input search query and displaying the results to the user.

[0480] The "emotion recognition means" is a means for analyzing and recognizing the user's emotional state.

[0481] An "emotion engine means" is a means for adjusting how information is presented based on the perceived emotional state of the user.

[0482] This invention describes a system for improving the shopping experience for brick-and-mortar stores that digitizes paper documents in the home and performs management, search, summarization, and information recommendation, in addition to a system that combines an emotion engine that recognizes user emotions.

[0483] System configuration and functions

[0484] This system has the following configuration and functions:

[0485] 1. Paper document capture method

[0486] This is a method for users to take a photo of a paper coupon or receipt using their smartphone camera and import it as a digital image. For example, a user can take a photo of a receipt they receive at a store cash register using a dedicated app.

[0487] 2. Character recognition method (OCR)

[0488] The server receives the digital image sent by the user and uses OCR technology to extract text information from the image. The Google Cloud Vision API is used to achieve highly accurate character recognition.

[0489] 3. Data Classification Methods

[0490] The server analyzes the extracted text and categorizes the data based on the type of coupon or receipt, using a machine learning algorithm to categorize the data into categories such as food, daily necessities, and services.

[0491] 4. Database Means

[0492] The server stores the classified data in the Firebase database and manages it as structured data, allowing users to quickly access the information they need.

[0493] 5. Search Input Methods

[0494] A way for a user to enter a search query into a search bar within an application, for example, "Show me this month's grocery receipts."

[0495] 6. Search result display method

[0496] It is a way for the server to search the Firebase database based on a search query and display the relevant data to the user, along with generating a summary of the search results.

[0497] 7. Information recommendation means

[0498] This is a means by which the server analyzes information based on the frequency and importance of the user's use and recommends appropriate coupons and products to the user. For example, it analyzes the user's purchase history and recommends coupons for products that the user is likely to purchase next.

[0499] 8. Emotion recognition means

[0500] The server uses technology to recognize the user's emotions. It uses Amazon Rekognition to analyze the user's emotional state from their typing speed and facial expressions.

[0501] 9. Emotional Engine Means

[0502] It is a way for the server to recommend information and tailor search result summaries based on the user's perceived emotions. The emotion engine can present simple, essential information when the user is stressed, and detailed information when the user is relaxed.

[0503] Specific examples

[0504] For example, if a user types "Show this month's grocery receipts" into a smartphone app, the following will happen:

[0505] 1. The user takes a photo of the receipt with their smartphone and uploads it to the app.

[0506] 2. The app sends the digital image to a server.

[0507] 3. The server uses the Google Cloud Vision API to extract text from the image.

[0508] 4. The server classifies the extracted text information into categories (food, daily necessities, etc.) and stores it in the Firebase database.

[0509] 5. The user uses a search input method to enter the search query "Show this month's grocery receipts."

[0510] 6. The server searches the database, provides the relevant receipt information in a search result display means, and generates a summary.

[0511] 7. Use Amazon Rekognition to analyze user sentiment and adjust how information is displayed using an emotion engine.

[0512] Prompt Sentence Examples

[0513] When the user enters "Show this month's food receipts," the server displays the corresponding receipt information, and can provide information according to the user's emotions.

[0514] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0515] Step 1:

[0516] The user takes a photo of a paper coupon or receipt with their smartphone.

[0517] Input: Paper coupons and receipts

[0518] Output: Coupons and receipts as digital images

[0519] How it works: The user takes a photo of a paper coupon or receipt using the camera function of the dedicated app. The captured image is saved to the smartphone's storage.

[0520] Step 2:

[0521] The terminal transmits the captured digital image to the server.

[0522] Input: The digital image acquired in step 1

[0523] Output: Digital image sent to server

[0524] How it works: The app sends the captured image to a cloud server, along with image metadata (such as the date and time of the photo and user information).

[0525] Step 3:

[0526] The server performs OCR processing to extract characters from the digital image.

[0527] Input: Digital image sent to the server

[0528] Output: Extracted text information

[0529] How it works: The server uses the Google Cloud Vision API to extract text from digital images, including preprocessing such as noise removal and deskewing.

[0530] Step 4:

[0531] The server classifies the extracted text information into categories.

[0532] Input: Character information extracted in step 3

[0533] Output: Classified text information

[0534] How it works: The server uses machine learning algorithms to classify the extracted text into categories such as coupons, receipts, food, and household items.

[0535] Step 5:

[0536] The server stores the classified character information in a database.

[0537] Input: Character information classified in step 4

[0538] Output: Text information stored in the database

[0539] How it works: The server stores classified textual information in the Firebase database. The stored data is managed as structured data, making it easy to search.

[0540] Step 6:

[0541] A user enters a search query into a search bar within the application.

[0542] Input: User's search query (e.g., "View this month's grocery receipts")

[0543] Output: The search query is sent to the server

[0544] What happens: A user types a query into an application's search bar, for example, "Show me this month's grocery receipts."

[0545] Step 7:

[0546] The server searches the database and retrieves the relevant data.

[0547] Input: The search query entered in step 6

[0548] Output: Data as search results

[0549] How it works: Your server searches your Firebase database to retrieve data that matches your query. For example, it aggregates relevant receipt information to display "Grocery Receipts of the Month."

[0550] Step 8:

[0551] The server generates and displays the search results and their summaries.

[0552] Input: Search results obtained in step 7

[0553] Output: Summarized search results

[0554] How it works: The server generates a summary based on the search results, for example aggregating multiple receipts and displaying the total amount and key purchase items.

[0555] Step 9:

[0556] The server recognizes the user's emotions and adjusts the display method according to that information.

[0557] Input: User's face image and input speed information

[0558] Output: How to display the adjusted information

[0559] How it works: The server uses Amazon Rekognition to analyze the user's facial image and typing speed to recognize their emotional state. For example, if the user is stressed, it will display simple, essential information, and if they are relaxed, it will provide detailed information.

[0560] By implementing the above steps, users can efficiently digitize and manage paper coupons and receipts, and quickly search and retrieve the information they need. Furthermore, the emotion engine makes it possible to provide personalized information based on the user's emotional state.

[0561] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0562] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0563] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0564] [Second embodiment]

[0565] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0566] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0567] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0568] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0569] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0570] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0571] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0572] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0573] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0574] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0575] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0576] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0577] As an embodiment of the present invention, a system for digitizing paper documents present in the home and managing, searching, summarizing, reporting, and recommending information will be described. This system has the following configuration and functions.

[0578] System configuration

[0579] 1. Paper document capture method

[0580] Users use a smartphone or scanner to take a photo of a paper document (utility bill, shopping receipt, medical bill, insurance document, school notice, pension newsletter, etc.) and capture it as a digital image.

[0581] 2. Character recognition method (OCR)

[0582] The server receives the digital image sent by the user and uses OCR technology to extract text information from the image. Character recognition is performed after preprocessing such as noise removal and skew correction.

[0583] 3. Data Classification Methods

[0584] The server analyzes the extracted text and classifies the data based on the document type (utility bill, receipt, medical statement, etc.) using machine learning algorithms.

[0585] 4. Database Means

[0586] The server stores the classified data in a database. The stored data is structured and managed in the form of pairs of item names and values.

[0587] 5. Search Input Methods

[0588] A user enters a search query into the application's search bar, for example, "electricity bill for the last 3 months."

[0589] 6. Search result display method

[0590] The server searches the database based on the search query and displays the relevant data to the user, optionally generating and displaying a summary of the search results.

[0591] 7. Information recommendation means

[0592] The server analyzes the information based on the frequency and importance of the user's use and recommends appropriate information to the user, such as notifying them that their insurance contract is due for renewal.

[0593] System operation example

[0594] 1. Utility bill management

[0595] Users take a photo of their electricity bill with their smartphone and upload it to a dedicated app.

[0596] The device sends the uploaded image to the server.

[0597] The server uses OCR technology to extract text from the image and classify it into the "utility bills" category.

[0598] The server stores the data in a database so that it can be quickly accessed when a user searches.

[0599] 2. Search for medical details

[0600] If a user wants to check the details of medical expenses for the past year, they enter "medical expenses for the past year" into the application.

[0601] The terminal sends a search query to the server, which searches the database for the relevant medical details.

[0602] The server displays the search results and their summaries to the user.

[0603] 3. Create monthly reports from receipts

[0604] If a user wants to know the total cost of food for the month, he / she inputs the request "Total cost of food for this month" into the application.

[0605] The terminal sends a request to the server, and the server aggregates the data for the specified period.

[0606] The server generates a report of the aggregated results and provides it to the user.

[0607] 4. Recommendation of insurance documents

[0608] The server analyzes the user's past search history and identifies important insurance-related information.

[0609] The server notifies the user that the time for renewal of the insurance contract is approaching and recommends related information to the user.

[0610] The user checks the recommended information in the app and takes the necessary action.

[0611] Summary

[0612] This invention allows users to efficiently digitize paper documents and quickly access the information contained therein. In particular, it allows users to manage their lives smartly and efficiently through search, summarization, reporting, and information recommendation functions. This system is expected to improve the quality of life by eliminating the complexity of managing paper documents for users and reducing time and stress.

[0613] The processing flow will be explained below.

[0614] Step 1:

[0615] Users take a photo of a paper document using a smartphone or scanner and import it as a digital image using a dedicated application.

[0616] Step 2:

[0617] The device sends the captured image to the server in an optimized format, compressing the image and adjusting the resolution to optimize the transfer speed.

[0618] Step 3:

[0619] The server temporarily stores the received digital images, along with the image's file format and metadata.

[0620] Step 4:

[0621] The server extracts text information from images using OCR (Optical Character Recognition) technology. Specifically, after preprocessing the image (noise removal, tilt correction, binarization, etc.), it detects text areas and recognizes the characters from them.

[0622] Step 5:

[0623] The server analyzes the extracted text and classifies the data based on the type of document, such as into categories like "utility bill," "receipt," or "medical statement," using machine learning algorithms.

[0624] Step 6:

[0625] The server organizes the classified data as structured data and stores it in a database. The data is stored in fields such as date, amount, and item name.

[0626] Step 7:

[0627] A user enters a search query through an application, for example, "electricity bill details for the last 3 months."

[0628] Step 8:

[0629] The terminal sends the user's search query to the server.

[0630] Step 9:

[0631] The server searches the database for data corresponding to the search query, using indexes to efficiently extract the data.

[0632] Step 10:

[0633] The server creates a summary of the search results using a summary generation algorithm as needed.

[0634] Step 11:

[0635] The server transmits the generated search results and summaries to the terminal.

[0636] Step 12:

[0637] The terminal displays search results and summaries to the user, allowing the user to quickly access the information they need.

[0638] Step 13:

[0639] The server analyzes the user's past search history, frequency of use, access patterns, etc. Based on this, it applies an algorithm to recommend important information and actions.

[0640] Step 14:

[0641] The server recommends information that it deems important to the user, and this information is sent to the device as a notification.

[0642] Step 15:

[0643] The terminal displays a notification of the recommended information to the user and provides an interface that allows the user to view detailed information.

[0644] These are the processing steps for a system that digitizes paper documents and performs management, search, summarization, reporting, and information recommendation, thereby simplifying paper document management for users and enabling quick access to the information they need.

[0645] Example 1

[0646] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0647] In today's world, homes are overflowing with paper documents, making it difficult to manage them efficiently. Manually searching through documents to find the information you need takes time and effort. It's also difficult to identify useful information based on its importance and frequency of use. There's a need for a system that can solve these issues and enable users to digitize paper documents and efficiently manage, search, summarize, report, and recommend information.

[0648] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0649] In this invention, the server includes a means for capturing paper documents as digital images, a means for converting the captured image data into text information, a means for classifying the text information and converting it into structured data, a data storage means, a means for a user to input a search query, a means for searching a data recording device based on the search query and displaying the results, a means for summarizing data based on the user's search query, and a means for recommending information based on user behavior data. This allows users to efficiently digitize paper documents and easily access the information. Furthermore, the system enables users to manage their lives smartly and efficiently through search, summarization, reporting, and information recommendations.

[0650] "Paper documents" refers to printed or written forms of paper, including electricity bills, shopping receipts, medical statements, insurance documents, school notices, pension newsletters, etc.

[0651] A "digital image" refers to an image file generated by photographing or scanning a physical paper document with a device such as a smartphone or scanner.

[0652] "Capture means" refers to a device and application software for capturing a paper document as a digital image.

[0653] "Character recognition means" refers to a technology that analyzes character information in an image and converts it into character data, and generally uses optical character recognition (OCR) technology.

[0654] "Data classification means" refers to algorithms and techniques for classifying extracted textual information into different categories (e.g., utility bills, receipts, medical statements, etc.).

[0655] "Data storage means" refers to a database system for long-term storage of information after data classification, which appropriately structures and stores digital data.

[0656] "Search input means" refers to an interface through which a user inputs a query to search for specific information, and sends this to a server via a terminal.

[0657] "Search result display means" refers to a function for displaying to a user data retrieved based on a search query, optionally providing results in a summarized form.

[0658] "Summary generation means" refers to the technology and algorithms used to summarize search results and information contained in a database and present it to the user in an easy-to-understand manner.

[0659] "Information recommendation means" refers to technology for recommending appropriate information and actions based on a user's usage history and behavioral data.

[0660] The system of the present invention is designed to efficiently digitize paper documents in the home and perform management, retrieval, summarization, reporting, and information recommendation. The structure of this system and its technical implementation are described in detail below.

[0661] System configuration and usage

[0662] Paper document capture method

[0663] Users use digital devices such as smartphones or scanners to capture digital images of paper documents (e.g., utility bills, receipts, medical statements, insurance documents, etc.) and can upload these digital images to the system through a dedicated application.

[0664] Character recognition means (OCR)

[0665] The digital image uploaded by the device is sent to a server, which then uses OCR technology, such as Amazon Textract or Google Cloud Vision, to extract text from the image. The OCR process includes preprocessing steps such as noise reduction and deskew.

[0666] Data Classification Methods

[0667] The server analyzes the extracted text and classifies the data based on the document type, using machine learning algorithms such as support vector machines (SVM) and random forests, for example, into categories such as medical statements, receipts, and utility bills.

[0668] Data storage means

[0669] The server stores the classified data in a database. The data is structured as a pair of item names and values. The database uses a database management system such as MySQL or PostgreSQL.

[0670] Search input method

[0671] The user enters a search query into the application's search bar. For example, the user might enter a query such as "electricity bill details for the past three months." The device then sends this search query to the server.

[0672] Search result display method

[0673] The server searches the database based on the received search query and extracts the relevant data. The search results are summarized and displayed to the user. The search function uses technologies such as Lucene and Elasticsearch.

[0674] Information recommendation means

[0675] The server analyzes the information based on the user's usage history and importance, and recommends appropriate information. For example, it could provide a function to notify users when their insurance contract is due for renewal.

[0676] Specific operation example

[0677] Utility bill management

[0678] Example prompt: "Please upload a photo of your electricity bill taken with your smartphone."

[0679] Users take a photo of their electricity bill with their smartphone and upload it to a dedicated app.

[0680] The device sends the uploaded image to the server.

[0681] The server uses OCR technology to extract text information from the image and classify it into the "utility bills" category.

[0682] The server stores the data in a database.

[0683] Search for medical details

[0684] Example prompt: "Please search for medical expense details for the past year."

[0685] A user types "medical expenses past year" into the app's search bar.

[0686] The device sends a search query to the server.

[0687] The server searches the database, extracts and summarizes the relevant medical details, and displays them to the user.

[0688] Generate monthly reports from receipts

[0689] Example prompt: "What is the total cost of food this month?"

[0690] The user enters "Total food expenses this month" into the app.

[0691] The device sends a query to the server.

[0692] The server aggregates the data for the specified period and generates a report.

[0693] The server sends the report to the user.

[0694] Recommendation of insurance documents

[0695] Example prompt: "I would like to receive insurance policy renewal notices."

[0696] The server analyzes the user's past search history and identifies important insurance-related information.

[0697] The server notifies the user that the time for renewal of the insurance contract is approaching and recommends related information to the user.

[0698] The user checks the recommended information in the app and takes the necessary action.

[0699] The system allows users to efficiently digitize paper documents and access information more quickly and easily, while search result summaries and information recommendations help users manage their lives more efficiently.

[0700] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0701] Step 1:

[0702] A user captures a paper document as a digital image using a smartphone or scanner.

[0703] Input: Paper document

[0704] Output: Digital image

[0705] Specific actions: The user launches the smartphone's camera app and takes a photo of a document, or scans a document using a scanner. The user then imports the captured or scanned image into the app.

[0706] Step 2:

[0707] The device uploads the digital images taken or scanned to a server via a dedicated app.

[0708] Input: Digital image

[0709] Output: Digital image sent to server

[0710] Specific operation: The user clicks the "Upload" button on the dedicated app to send the photographed or scanned image data to the cloud server. The device displays the sending status to the user.

[0711] Step 3:

[0712] The server performs OCR processing on the received digital image to extract text information.

[0713] Input: Digital image

[0714] Output: Extracted text information

[0715] How it works: The server analyzes the received image data using OCR technology such as Amazon Textract or Google Cloud Vision. The OCR process includes noise removal and tilt correction, and the character information in the image is extracted as text data.

[0716] Step 4:

[0717] The server analyzes the extracted text information and categorizes it based on the document type.

[0718] Input: Extracted text information

[0719] Output: Categorized data

[0720] How it works: The server analyzes text information using machine learning algorithms such as SVM and Random Forest. For example, the server identifies the keyword "electricity bill" and classifies it into the "utility bill" category.

[0721] Step 5:

[0722] The server stores the classified data in a database.

[0723] Input: Categorical data

[0724] Output: Data stored in the database

[0725] How it works: The server uses a database management system such as MySQL or PostgreSQL to store data categorized into categories in the form of item name / value pairs. For example, data in the "Utility Expenses" category is recorded in the form of "Date," "Amount," "Category," etc.

[0726] Step 6:

[0727] A user enters a search query into the search bar of an application, and the terminal sends the search query to a server.

[0728] Input: search query

[0729] Output: The search query sent to the server

[0730] What happens: The user enters "electricity bill details for the last 3 months" into the application's search bar and clicks the search button. The device sends the search query to the server.

[0731] Step 7:

[0732] The server searches the database based on the received search query, extracts and summarizes the relevant data.

[0733] Input: search query

[0734] Output: Summarized search results

[0735] What it does: The server uses technologies such as Lucene or Elasticsearch to search the database, extracts relevant data, and optionally summarizes the search results using an automatic summarization algorithm.

[0736] Step 8:

[0737] The server sends the summarized search results to the terminal, which displays the results to the user.

[0738] Input: Summarized search results

[0739] Output: The result displayed to the user

[0740] Specific operation: The server sends search results to the device in JSON format, etc. The device displays the received results in an easy-to-read format within the application interface, and the user confirms the search results.

[0741] Step 9:

[0742] The server analyzes the information based on the frequency of use and importance of the user, and recommends appropriate information to the user.

[0743] Input: User behavior data and usage history

[0744] Output: Recommendation notification

[0745] What happens: The server uses machine learning algorithms to analyze user behavior data. For example, it identifies when an insurance contract is due for renewal and notifies the user. The user then checks the recommendations in the app and takes the necessary action.

[0746] (Application example 1)

[0747] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0748] In recent years, the number of users of online shopping sites has increased, increasing the need for efficient management of purchase history and order details. However, managing paper-based statements is time-consuming, and it is not easy to search, summarize, or recommend information. In particular, it is difficult to summarize past purchase history or accurately recommend the next purchase date or related products. For this reason, there is a need for a system that improves user convenience and enables efficient data management and information provision.

[0749] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0750] In this invention, the server includes means for capturing paper documents as digital images, character recognition means for converting the captured image data into character information, data classification means for classifying the character information and converting it into structured data, database means for saving the data converted by the data classification means, search input means for a user to input a search query, search result display means for searching the database based on the search query and displaying the results, and information recommendation means for processing the user's purchase history and recommending the next purchase date and related products. This allows the user to efficiently manage their purchase history and order details and quickly receive searches, summaries, and recommendations.

[0751] "Means for capturing paper documents as digital images" refers to a function that allows a user to photograph a paper document using a smartphone or scanner and capture the image as a digital image.

[0752] "Character recognition means for converting captured image data into character information" is a function that uses OCR technology to extract character information from digital images.

[0753] The "data classification means for classifying character information and converting it into structured data" is a function that analyzes extracted character information, classifies the data based on the type and content of the document, and converts it into structured data in the form of item name and value pairs.

[0754] The "database means for storing data converted by the data classification means" is a function for storing structured data in a database so that it can be accessed efficiently later.

[0755] The "search input means by which a user inputs a search query" refers to a function by which a user inputs a search query to search for specific information in a search bar of an application, for example.

[0756] The "search result display means for searching a database based on a search query and displaying the results" is a function for retrieving data corresponding to the search query from a database and displaying the data to the user.

[0757] The "information recommendation means for processing the user's purchase history and recommending the next purchase time and related products" is a function that analyzes the user's past purchase history and recommends the next appropriate purchase time and related products.

[0758] As an embodiment of the present invention, we will explain a system that efficiently manages purchase histories and order details on an online shopping site and performs searches, summarization, and recommendations. This system has the following configuration and functions.

[0759] System configuration

[0760] 1. Paper document capture method

[0761] Users use their smartphone camera or a dedicated scanner to take a picture of the purchase receipt from the online shopping site and import it as a digital image.

[0762] 2. Character recognition method (OCR)

[0763] The server receives the digital image sent by the user and extracts text information from the image using OCR technology (such as pytesseract), as well as preprocessing the image by grayscale conversion and noise reduction.

[0764] 3. Data Classification Methods

[0765] The server analyzes the extracted text and classifies the data based on the document type (food, home appliances, clothing, etc.) and content using machine learning algorithms.

[0766] 4. Database Means

[0767] The server stores the classified data in a database. The stored data is structured and managed in the form of pairs of item names and values.

[0768] 5. Search Input Methods

[0769] A user types a search query into the search bar of a smartphone application, for example, "amount spent on food in the last 3 months."

[0770] 6. Search result display method

[0771] The server searches the database based on the search query and displays the relevant data to the user, optionally generating and displaying a summary of the search results.

[0772] 7. Information recommendation means

[0773] The server processes the user's purchase history and recommends the next purchase time and related products, for example, notifying the user that it is almost time to buy detergent again.

[0774] System operation example

[0775] 1. Managing purchase details

[0776] Users take a photo of their purchase details with their smartphone and upload it to a dedicated app. The device sends the image to a server, which converts it into text using OCR technology. The server then categorizes the information and organizes it into categories such as "food" or "home appliances."

[0777] 2. Search for food purchase amounts

[0778] If a user wants to find out how much they spent on food over the past three months, they simply enter "food spending over the past three months" into the application. The device sends the search query to the server, which then searches the database for the relevant data, summarizes it, and displays it.

[0779] 3. Recommendation for next purchase

[0780] Users receive recommendations for their next purchase and related products based on their purchase history. For example, if detergent stocks are running low, the server will send a notification recommending the next purchase.

[0781] Prompt Sentence Examples

[0782] Example 1: Want to know how much you spent on food over the past three months?

[0783] Input: "Food purchases for the past 3 months"

[0784] Output: "I've spent ¥5000 on food in the past three months."

[0785] Example 2: Recommending your next detergent purchase

[0786] Input: "Next recommended time to purchase detergent"

[0787] Output: "Your detergent stock is running low, so we recommend you buy more next time."

[0788] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0789] Step 1:

[0790] The user takes a photo of the purchase receipt with their smartphone and uploads the image to a dedicated app.

[0791] Input: Image of purchase details taken with a smartphone

[0792] Output: Digital image file

[0793] Specific operation: Use the smartphone camera to take a photo of a paper purchase invoice and upload the image to the server from the application.

[0794] Step 2:

[0795] The server applies OCR technology to convert the received image data into text information.

[0796] Input: Digital images uploaded by users

[0797] Output: Extracted text information

[0798] Specific operation: The server uses an image processing library (e.g., OpenCV) to convert the image to grayscale and perform noise reduction, then performs OCR using pytesseract.

[0799] Step 3:

[0800] The server analyzes the character information extracted by OCR, classifies the data, and structures it.

[0801] Input: Character information extracted by OCR

[0802] Output: Classified structured data

[0803] Specific operation: The server uses a machine learning algorithm to analyze the text information, classify the data by category or item, such as "food" or "home appliances," and structure it in the form of item name / value pairs.

[0804] Step 4:

[0805] The server stores the structured data in a database.

[0806] Input: Categorized and structured data

[0807] Output: Data stored in the database

[0808] Specific operation: The server connects to a database management system (e.g., MySQL) and stores the classified data by item.

[0809] Step 5:

[0810] A user types a search query into the search bar of a smartphone app.

[0811] Input: A search query such as "food spending in the last 3 months"

[0812] Output: None (because the query is sent to the server)

[0813] What happens: A user enters a search query into the application's search bar, and the query is sent to the server.

[0814] Step 6:

[0815] The server searches the database based on the search query and retrieves the relevant data.

[0816] Input: Search query

[0817] Output: Search results

[0818] What happens: The server runs an SQL query against the database to retrieve the relevant data, for example, the food purchase history for the past three months.

[0819] Step 7:

[0820] The server summarizes the retrieved data and displays it to the user.

[0821] Input: Search results

[0822] Output: Search results displayed as a summary

[0823] Specific operation: The server aggregates and analyzes the acquired data and organizes it as a summary. For example, it calculates the total amount of food purchases for the past three months and displays it to the user.

[0824] Step 8:

[0825] The server analyzes the user's purchasing history and recommends the next purchase time and related products.

[0826] Input: User purchase history data

[0827] Output: Recommendation

[0828] How it works: The server uses a recommendation algorithm to analyze the user's purchase history and generate recommendations for the next appropriate purchase time and related products. For example, if detergent stock is low, a notification will be sent recommending the next purchase.

[0829] Prompt Sentence Examples

[0830] Example 1: Want to know how much you spent on food over the past three months?

[0831] Input: "Food purchases for the past 3 months"

[0832] Output: "I've spent ¥5000 on food in the past three months."

[0833] Example 2: Recommending your next detergent purchase

[0834] Input: "Next recommended time to purchase detergent"

[0835] Output: "Your detergent stock is running low, so we recommend you buy more next time."

[0836] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0837] As an embodiment of the present invention, we will explain a system that digitizes paper documents present in the home and performs management, search, summarization, reporting, and information recommendation, as well as a system that combines an emotion engine that recognizes user emotions. This system has the following configuration and functions.

[0838] System configuration

[0839] 1. Paper document capture method

[0840] Users use a smartphone or scanner to take a photo of a paper document (utility bill, shopping receipt, medical bill, insurance document, school notice, pension newsletter, etc.) and capture it as a digital image.

[0841] 2. Character recognition method (OCR)

[0842] The server receives the digital image sent by the user and uses OCR technology to extract text information from the image. Character recognition is performed after preprocessing such as noise removal and skew correction.

[0843] 3. Data Classification Methods

[0844] The server analyzes the extracted text and classifies the data based on the document type (utility bill, receipt, medical statement, etc.) using machine learning algorithms.

[0845] 4. Database Means

[0846] The server stores the classified data in a database. The stored data is structured and managed in the form of pairs of item names and values.

[0847] 5. Search Input Methods

[0848] A user enters a search query into the application's search bar, for example, "electricity bill for the last 3 months."

[0849] 6. Search result display method

[0850] The server searches the database based on the search query and displays the relevant data to the user, optionally generating and displaying a summary of the search results.

[0851] 7. Information recommendation means

[0852] The server analyzes the information based on the frequency and importance of the user's use and recommends appropriate information to the user, such as notifying them that their insurance contract is due for renewal.

[0853] 8. Emotion recognition means

[0854] The server uses technology to recognize the user's emotions, for example, by analyzing the user's emotional state from their input speed, context, facial recognition, etc.

[0855] 9. Emotion Engine

[0856] The server then recommends information and adjusts the summary of search results based on the user's recognized emotions. The emotion engine changes its behavior depending on the user's emotional state, for example, displaying simple but important information when the user is stressed, or providing detailed information when the user is relaxed.

[0857] System operation example

[0858] 1. Utility bill management

[0859] Users take a photo of their electricity bill with their smartphone and upload it to a dedicated app.

[0860] The device sends the uploaded image to the server.

[0861] The server uses OCR technology to extract text from the image and classify it into the "utility bills" category.

[0862] The server stores the data in a database so that it can be quickly accessed when a user searches.

[0863] 2. Search for medical details

[0864] If a user wants to check the details of medical expenses for the past year, they enter "medical expenses for the past year" into the application.

[0865] The terminal sends a search query to the server, which searches the database for the relevant medical details.

[0866] The server displays the search results and their summaries to the user.

[0867] 3. Create monthly reports from receipts

[0868] If a user wants to know the total cost of food for the month, he / she inputs the request "Total cost of food for this month" into the application.

[0869] The terminal sends a request to the server, and the server aggregates the data for the specified period.

[0870] The server generates a report of the aggregated results and provides it to the user.

[0871] 4. Recommendation of insurance documents

[0872] The server analyzes the user's past search history and identifies important insurance-related information.

[0873] The server notifies the user that the time for renewal of the insurance contract is approaching and recommends related information to the user.

[0874] The user checks the recommended information in the app and takes the necessary action.

[0875] 5. Emotion-based information presentation

[0876] The server recognizes the user's emotions, and if the user is feeling stressed, for example, important and concise information is displayed preferentially.

[0877] If you're relaxed, it will show you more information and additional options.

[0878] This allows for optimal information provision according to the user's emotional state.

[0879] Summary

[0880] This invention allows users to efficiently digitize paper documents and quickly access the information they hold. In particular, by incorporating search, summarization, reporting, and information recommendation functions, as well as taking into account the user's emotions, it is possible to provide more personalized information. This system is expected to improve users' quality of life by eliminating the hassle of managing paper documents and reducing the time and stress involved.

[0881] The processing flow will be explained below.

[0882] Step 1:

[0883] Users take a photo of a paper document using a smartphone or scanner and import it as a digital image using a dedicated application.

[0884] Step 2:

[0885] The device sends the captured image to the server in an optimized format, compressing the image and adjusting the resolution to optimize the transfer speed.

[0886] Step 3:

[0887] The server temporarily stores the received digital images, along with the image's file format and metadata.

[0888] Step 4:

[0889] The server extracts text information from images using OCR (Optical Character Recognition) technology. Specifically, after preprocessing the image (noise removal, tilt correction, binarization, etc.), it detects text areas and recognizes the characters from them.

[0890] Step 5:

[0891] The server analyzes the extracted text and classifies the data based on the type of document, such as into categories like "utility bill," "receipt," or "medical statement," using machine learning algorithms.

[0892] Step 6:

[0893] The server organizes the classified data as structured data and stores it in a database. The data is stored in fields such as date, amount, and item name.

[0894] Step 7:

[0895] A user enters a search query through an application, for example, "electricity bill details for the last 3 months."

[0896] Step 8:

[0897] The terminal sends the user's search query to the server.

[0898] Step 9:

[0899] The server searches the database for data corresponding to the search query, using indexes to efficiently extract the data.

[0900] Step 10:

[0901] The server creates a summary of the search results using a summary generation algorithm as needed.

[0902] Step 11:

[0903] The server transmits the generated search results and summaries to the terminal.

[0904] Step 12:

[0905] The terminal displays search results and summaries to the user, allowing the user to quickly access the information they need.

[0906] Step 13:

[0907] The server analyzes the user's past search history, frequency of use, access patterns, etc. Based on this, it applies an algorithm to recommend important information and actions.

[0908] Step 14:

[0909] The server recommends information that it deems important to the user, and this information is sent to the device as a notification.

[0910] Step 15:

[0911] The terminal displays a notification of the recommended information to the user and provides an interface that allows the user to view detailed information.

[0912] Step 16:

[0913] The server uses technology to recognize the user's emotions, for example, by analyzing the user's emotional state from their typing speed, context, facial recognition, etc.

[0914] Step 17:

[0915] The server recommends information and adjusts the summary of search results based on the user's recognized emotions. The emotion engine displays simple, essential information when the user is stressed, and provides detailed information when the user is relaxed.

[0916] Step 18:

[0917] The device displays the most appropriate information according to the user's emotional state, thereby realizing personalized information provision that matches the user's emotional state.

[0918] These are the processing steps of the system that combines the emotion engine. This system simplifies the management of paper documents for users, enables quick access to necessary information, and provides information based on emotions.

[0919] Example 2

[0920] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0921] In conventional paper document management systems, digitizing and searching documents is cumbersome, and the accuracy of summarization and information recommendations is low.In addition, information presentation does not take into account the user's emotional state, and the user experience is not optimized.

[0922] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0923] In this invention, the server includes means for capturing paper documents as digital images, character recognition means for converting the captured image data into character information, data classification means for classifying the character information and converting it into structured data, database means for saving the data converted by the data classification means, search input means for a user to input a search query, search result display means for searching the database based on the search query and displaying the results, and emotion recognition means for recognizing the user's emotion and adjusting the way information is presented. This allows users to efficiently digitize documents, easily search and acquire information, and receive information optimally suited to their emotional state.

[0924] "Means for capturing paper documents as digital images" refers to a device or software that allows a user to photograph a paper document using a smartphone or scanner and capture the image as a digital image.

[0925] "Character recognition means" refers to a technology for extracting character information from a digital image, specifically using OCR (optical character recognition) technology.

[0926] A "data classification means" is an algorithm or machine learning model that analyzes extracted text information and classifies it into specific categories.

[0927] "Database means" refers to a database system for structuring and storing classified data, including SQL and NoSQL databases.

[0928] A "search input means" is an application or system that provides an interface for a user to enter a search query in order to search for specific information.

[0929] The "search result display means" is an interface for displaying to the user the data acquired based on the search query, and displays it as text or graphs on the screen.

[0930] "Emotion recognition means" is a technology for analyzing the user's emotional state, and recognizes emotions using data such as input speed, context, and facial recognition.

[0931] MODE FOR CARRYING OUT THE INVENTION

[0932] The system of the present invention efficiently digitizes paper documents kept at home by users and manages, searches, summarizes, reports, and recommends information. It also includes an emotion engine that recognizes the user's emotional state and provides information accordingly. This system operates using the following hardware and software:

[0933] Hardware and Software Configuration

[0934] 1. Paper document capture method

[0935] Users use a smartphone or scanner to capture digital images of paper documents (such as electricity bills, shopping receipts, medical bills, etc.), and then upload these images to the device using a dedicated application.

[0936] 2. Character recognition means (OCR technology)

[0937] The server receives the digital image sent by the user and uses OCR (Optical Character Recognition) technology to extract text information from the image. Specifically, a library such as Tesseract OCR is used to perform preprocessing such as noise removal and tilt correction before recognizing the text.

[0938] 3. Data Classification Methods

[0939] The server analyzes the text extracted by OCR and determines the document type using machine learning algorithms such as SVM (support vector machine) and neural networks. For example, an electricity bill would be classified as a "utility bill" category.

[0940] 4. Database Means

[0941] The server stores the classified data in a database using a structured database such as MySQL or PostgreSQL, where the data is stored in the form of field name / value pairs, allowing for fast and efficient data searches and report generation.

[0942] 5. Search Input Methods

[0943] The user enters a search query into the application's search bar, for example, a specific query such as "electricity bill details for the last three months." This search query is sent to the server via the device.

[0944] 6. Search result display method

[0945] The server searches the database based on the received search query and extracts the relevant data. It then generates a summary from the extracted data and displays the results on the user's device. For example, a search result for "electricity bill details for the past three months" would display the details for each month and the total amount.

[0946] 7. Information recommendation means

[0947] The server recommends appropriate information based on the user's usage history and importance. For example, if an insurance contract is due for renewal, the server notifies the user of this information and recommends detailed information. This ensures that the user does not miss important information.

[0948] 8. Emotion recognition means

[0949] The server analyzes the user's emotional state based on their input speed, context, facial recognition, etc. For example, it can accurately recognize the user's emotional state by using Python's OpenCV library and various cloud APIs (e.g., Azure's Emotion API). If the input speed is slow or if words indicating stress are detected in the context, it is determined that the user is feeling stressed.

[0950] 9. Emotion Engine

[0951] The server adjusts the way information is presented based on the user's recognized emotions. For example, if the user is stressed, only simple and important information is displayed, while if the user is relaxed, detailed information and additional options are provided. This allows the server to provide optimal information according to the user's emotional state.

[0952] Examples of concrete examples and prompts

[0953] Example 1: Utility bill management

[0954] Users take a photo of their electricity bill with their smartphone and upload it to a dedicated app.

[0955] The device sends the uploaded image to the server.

[0956] The server uses OCR technology to extract text from the image and classify it into the "utility bills" category.

[0957] The server stores the data in a database so that it can be quickly accessed when a user searches.

[0958] Example 2: Searching for medical details

[0959] If a user wants to check the details of medical expenses for the past year, they enter the search query "medical expenses for the past year."

[0960] The terminal sends a search query to the server, which searches the database for the relevant medical details.

[0961] The server displays the search results and their summaries to the user.

[0962] Example 3: Creating a monthly report from receipts

[0963] If the user wants to know the total food expenses for the month, he / she inputs a request such as "Total food expenses for this month."

[0964] The terminal sends a request to the server, and the server aggregates the data for the specified period.

[0965] The server generates a report of the aggregated results and provides it to the user.

[0966] Example 4: Recommending insurance documents

[0967] The server analyzes the user's past search history and identifies important insurance-related information.

[0968] The server notifies the user that the time for renewal of the insurance contract is approaching and recommends related information to the user.

[0969] The user reviews the recommended information and takes the necessary action.

[0970] Example 5: Emotion-based information provision

[0971] The server recognizes the user's emotions, and if the user is feeling stressed, for example, it prioritizes displaying concise and important information.

[0972] If the user is relaxed, provide more information or additional options, which provides optimal information to the user.

[0973] Example prompts to input to the generative AI model

[0974] Please tell me your electricity bill details for the past three months.

[0975] "Calculate the total cost of food last month."

[0976] Please show me your medical expenses for the past year.

[0977] "Can you tell me when my insurance policy is due for renewal?"

[0978] "Show me recommendations based on my current mood."

[0979] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0980] Step 1:

[0981] The user takes a photo of the document and captures it as a digital image.

[0982] A user takes a photo of a paper document using a smartphone or scanner. For example, they take a photo of an electricity bill. Then, they use a dedicated application to upload the captured image file to their device. The input is a paper document, and the output is a digital image file.

[0983] Step 2:

[0984] The device sends the digital image to the server

[0985] The terminal transfers the digital image file uploaded by the user to the server. The input is the digital image file, and the output is the digital image file stored on the server. This completes the communication until the image reaches the server.

[0986] Step 3:

[0987] The server extracts text information using OCR technology.

[0988] The server analyzes the received digital image and extracts text information from the image using OCR technology. Specifically, it uses the Tesseract OCR library and performs preprocessing such as noise removal and tilt correction. The input is the digital image file, and the output is the extracted text information.

[0989] Step 4:

[0990] The server classifies the data

[0991] The server analyzes the text extracted by OCR and uses machine learning algorithms (such as SVM or neural networks) to determine the type of document and classify it. For example, an electricity bill is classified into the "utility bill" category. The input is text, and the output is classified data.

[0992] Step 5:

[0993] The server stores the classified data in a database.

[0994] The server stores the classified data in a database. Specifically, it uses a MySQL or PostgreSQL database and stores the data in a structured format of field name / value pairs. The input is the classified data, and the output is the data stored in the database.

[0995] Step 6:

[0996] The user enters a search query

[0997] A user enters a query into the application's search bar to search for specific information. For example, a specific search query such as "electricity bill details for the last three months." The input is the user's search query, and the output is the transmission of the search query from the device to the server.

[0998] Step 7:

[0999] The server searches the database and displays the results

[1000] The server searches the database based on the received search query and extracts the relevant data. A summary is generated from the extracted data and the results are displayed on the user's device. For example, a search result for "electricity bill details for the past three months" would display the details for each month and the total amount. The input is the search query, and the output is the display of the search results.

[1001] Step 8:

[1002] Server recommends information

[1003] The server recommends appropriate information based on the user's usage history and importance. For example, if an insurance contract is due for renewal, the server notifies the user of this information and recommends detailed information. This prevents the user from missing important information. The input is the user's usage history and importance, and the output is a notification of recommended information.

[1004] Step 9:

[1005] The server recognizes emotions and adjusts information

[1006] The server analyzes and recognizes the user's emotional state based on their input speed, context, and facial recognition. Specifically, it uses the Python OpenCV library and Azure's Emotion API. For example, if the input speed is slow or if the user uses many words indicating stress, it determines that the user is feeling stressed. The input is the user's input data and video data, and the output is the analysis result of the user's emotional state.

[1007] Step 10:

[1008] The server provides information using an emotion engine

[1009] The server adjusts the way information is presented based on the user's recognized emotions. For example, if the user is stressed, it displays only simple and important information, while if the user is relaxed, it provides detailed information and additional options. The input is the analysis result of the user's emotional state, and the output is the adjusted information provided.

[1010] (Application example 2)

[1011] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1012] In conventional paper document management systems, managing paper coupons and receipts is cumbersome, making it difficult to digitize them for easy user use. Furthermore, providing information according to the user's emotional state is not considered, which does not lead to an improved user experience. This has led to a demand for a system that allows users to efficiently manage paper information and quickly obtain the information they need without feeling stressed.

[1013] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1014] In this invention, the server includes means for capturing paper documents as digital images, character recognition means for converting the captured image data into character information, data classification means for classifying the character information and converting it into structured data, database means for saving the data converted by the data classification means, search input means for a user to input a search query, search result display means for searching the database based on the search query and displaying the results, emotion recognition means for recognizing the emotional state of the user, and emotion engine means for adjusting the method of providing information based on the recognized emotional state. This enables users to efficiently digitize and manage paper coupons and receipts, and further enables providing information according to the user's emotional state, thereby providing a more comfortable and personalized user experience.

[1015] - "Means for capturing paper documents" means means for capturing paper documents as digital images.

[1016] The "character recognition means" is a means for converting captured digital image data into character information.

[1017] The "data classification means" is a means for classifying the character information converted by the character recognition means into categories and converting it into structured data.

[1018] "Database Means" means a means for storing classified structured data and making it accessible as needed.

[1019] A "search input means" is a means by which a user inputs a search query.

[1020] The "search result display means" is a means for searching a database based on an input search query and displaying the results to the user.

[1021] The "emotion recognition means" is a means for analyzing and recognizing the user's emotional state.

[1022] An "emotion engine means" is a means for adjusting how information is presented based on the perceived emotional state of the user.

[1023] This invention describes a system for improving the shopping experience for brick-and-mortar stores that digitizes paper documents in the home and performs management, search, summarization, and information recommendation, in addition to a system that combines an emotion engine that recognizes user emotions.

[1024] System configuration and functions

[1025] This system has the following configuration and functions:

[1026] 1. Paper document capture method

[1027] This is a method for users to take a photo of a paper coupon or receipt using their smartphone camera and import it as a digital image. For example, a user can take a photo of a receipt they receive at a store cash register using a dedicated app.

[1028] 2. Character recognition method (OCR)

[1029] The server receives the digital image sent by the user and uses OCR technology to extract text information from the image. The Google Cloud Vision API is used to achieve highly accurate character recognition.

[1030] 3. Data Classification Methods

[1031] The server analyzes the extracted text and categorizes the data based on the type of coupon or receipt, using a machine learning algorithm to categorize the data into categories such as food, daily necessities, and services.

[1032] 4. Database Means

[1033] The server stores the classified data in the Firebase database and manages it as structured data, allowing users to quickly access the information they need.

[1034] 5. Search Input Methods

[1035] A way for a user to enter a search query into a search bar within an application, for example, "Show me this month's grocery receipts."

[1036] 6. Search result display method

[1037] It is a way for the server to search the Firebase database based on a search query and display the relevant data to the user, along with generating a summary of the search results.

[1038] 7. Information recommendation means

[1039] This is a means by which the server analyzes information based on the frequency and importance of the user's use and recommends appropriate coupons and products to the user. For example, it analyzes the user's purchase history and recommends coupons for products that the user is likely to purchase next.

[1040] 8. Emotion recognition means

[1041] The server uses technology to recognize the user's emotions. It uses Amazon Rekognition to analyze the user's emotional state from their typing speed and facial expressions.

[1042] 9. Emotional Engine Means

[1043] It is a way for the server to recommend information and tailor search result summaries based on the user's perceived emotions. The emotion engine can present simple, essential information when the user is stressed, and detailed information when the user is relaxed.

[1044] Specific examples

[1045] For example, if a user types "Show this month's grocery receipts" into a smartphone app, the following will happen:

[1046] 1. The user takes a photo of the receipt with their smartphone and uploads it to the app.

[1047] 2. The app sends the digital image to a server.

[1048] 3. The server uses the Google Cloud Vision API to extract text from the image.

[1049] 4. The server classifies the extracted text information into categories (food, daily necessities, etc.) and stores it in the Firebase database.

[1050] 5. The user uses a search input method to enter the search query "Show this month's grocery receipts."

[1051] 6. The server searches the database, provides the relevant receipt information in a search result display means, and generates a summary.

[1052] 7. Use Amazon Rekognition to analyze user sentiment and adjust how information is displayed using an emotion engine.

[1053] Prompt Sentence Examples

[1054] When the user enters "Show this month's food receipts," the server displays the corresponding receipt information, and can provide information according to the user's emotions.

[1055] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1056] Step 1:

[1057] The user takes a photo of a paper coupon or receipt with their smartphone.

[1058] Input: Paper coupons and receipts

[1059] Output: Coupons and receipts as digital images

[1060] How it works: The user takes a photo of a paper coupon or receipt using the camera function of the dedicated app. The captured image is saved to the smartphone's storage.

[1061] Step 2:

[1062] The terminal transmits the captured digital image to the server.

[1063] Input: The digital image acquired in step 1

[1064] Output: Digital image sent to server

[1065] How it works: The app sends the captured image to a cloud server, along with image metadata (such as the date and time of the photo and user information).

[1066] Step 3:

[1067] The server performs OCR processing to extract characters from the digital image.

[1068] Input: Digital image sent to the server

[1069] Output: Extracted text information

[1070] How it works: The server uses the Google Cloud Vision API to extract text from digital images, including preprocessing such as noise removal and deskewing.

[1071] Step 4:

[1072] The server classifies the extracted text information into categories.

[1073] Input: Character information extracted in step 3

[1074] Output: Classified text information

[1075] How it works: The server uses machine learning algorithms to classify the extracted text into categories such as coupons, receipts, food, and household items.

[1076] Step 5:

[1077] The server stores the classified character information in a database.

[1078] Input: Character information classified in step 4

[1079] Output: Text information stored in the database

[1080] How it works: The server stores classified textual information in the Firebase database. The stored data is managed as structured data, making it easy to search.

[1081] Step 6:

[1082] A user enters a search query into a search bar within the application.

[1083] Input: User's search query (e.g., "View this month's grocery receipts")

[1084] Output: The search query is sent to the server

[1085] What happens: A user types a query into an application's search bar, for example, "Show me this month's grocery receipts."

[1086] Step 7:

[1087] The server searches the database and retrieves the relevant data.

[1088] Input: The search query entered in step 6

[1089] Output: Data as search results

[1090] How it works: Your server searches your Firebase database to retrieve data that matches your query. For example, it aggregates relevant receipt information to display "Grocery Receipts of the Month."

[1091] Step 8:

[1092] The server generates and displays the search results and their summaries.

[1093] Input: Search results obtained in step 7

[1094] Output: Summarized search results

[1095] How it works: The server generates a summary based on the search results, for example aggregating multiple receipts and displaying the total amount and key purchase items.

[1096] Step 9:

[1097] The server recognizes the user's emotions and adjusts the display method according to that information.

[1098] Input: User's face image and input speed information

[1099] Output: How to display the adjusted information

[1100] How it works: The server uses Amazon Rekognition to analyze the user's facial image and typing speed to recognize their emotional state. For example, if the user is stressed, it will display simple, essential information, and if they are relaxed, it will provide detailed information.

[1101] By implementing the above steps, users can efficiently digitize and manage paper coupons and receipts, and quickly search and retrieve the information they need. Furthermore, the emotion engine makes it possible to provide personalized information based on the user's emotional state.

[1102] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1103] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1104] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1105] [Third embodiment]

[1106] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1107] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1108] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1109] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1110] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1111] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1112] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1113] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1114] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1115] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1116] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1117] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1118] As an embodiment of the present invention, a system for digitizing paper documents present in the home and managing, searching, summarizing, reporting, and recommending information will be described. This system has the following configuration and functions.

[1119] System configuration

[1120] 1. Paper document capture method

[1121] Users use a smartphone or scanner to take a photo of a paper document (utility bill, shopping receipt, medical bill, insurance document, school notice, pension newsletter, etc.) and capture it as a digital image.

[1122] 2. Character recognition method (OCR)

[1123] The server receives the digital image sent by the user and uses OCR technology to extract text information from the image. Character recognition is performed after preprocessing such as noise removal and skew correction.

[1124] 3. Data Classification Methods

[1125] The server analyzes the extracted text and classifies the data based on the document type (utility bill, receipt, medical statement, etc.) using machine learning algorithms.

[1126] 4. Database Means

[1127] The server stores the classified data in a database. The stored data is structured and managed in the form of pairs of item names and values.

[1128] 5. Search Input Methods

[1129] A user enters a search query into the application's search bar, for example, "electricity bill for the last 3 months."

[1130] 6. Search result display method

[1131] The server searches the database based on the search query and displays the relevant data to the user, optionally generating and displaying a summary of the search results.

[1132] 7. Information recommendation means

[1133] The server analyzes the information based on the frequency and importance of the user's use and recommends appropriate information to the user, such as notifying them that their insurance contract is due for renewal.

[1134] System operation example

[1135] 1. Utility bill management

[1136] Users take a photo of their electricity bill with their smartphone and upload it to a dedicated app.

[1137] The device sends the uploaded image to the server.

[1138] The server uses OCR technology to extract text from the image and classify it into the "utility bills" category.

[1139] The server stores the data in a database so that it can be quickly accessed when a user searches.

[1140] 2. Search for medical details

[1141] If a user wants to check the details of medical expenses for the past year, they enter "medical expenses for the past year" into the application.

[1142] The terminal sends a search query to the server, which searches the database for the relevant medical details.

[1143] The server displays the search results and their summaries to the user.

[1144] 3. Create monthly reports from receipts

[1145] If a user wants to know the total cost of food for the month, he / she inputs the request "Total cost of food for this month" into the application.

[1146] The terminal sends a request to the server, and the server aggregates the data for the specified period.

[1147] The server generates a report of the aggregated results and provides it to the user.

[1148] 4. Recommendation of insurance documents

[1149] The server analyzes the user's past search history and identifies important insurance-related information.

[1150] The server notifies the user that the time for renewal of the insurance contract is approaching and recommends related information to the user.

[1151] The user checks the recommended information in the app and takes the necessary action.

[1152] Summary

[1153] This invention allows users to efficiently digitize paper documents and quickly access the information contained therein. In particular, it allows users to manage their lives smartly and efficiently through search, summarization, reporting, and information recommendation functions. This system is expected to improve the quality of life by eliminating the complexity of managing paper documents for users and reducing time and stress.

[1154] The processing flow will be explained below.

[1155] Step 1:

[1156] Users take a photo of a paper document using a smartphone or scanner and import it as a digital image using a dedicated application.

[1157] Step 2:

[1158] The device sends the captured image to the server in an optimized format, compressing the image and adjusting the resolution to optimize the transfer speed.

[1159] Step 3:

[1160] The server temporarily stores the received digital images, along with the image's file format and metadata.

[1161] Step 4:

[1162] The server extracts text information from images using OCR (Optical Character Recognition) technology. Specifically, after preprocessing the image (noise removal, tilt correction, binarization, etc.), it detects text areas and recognizes the characters from them.

[1163] Step 5:

[1164] The server analyzes the extracted text and classifies the data based on the type of document, such as into categories like "utility bill," "receipt," or "medical statement," using machine learning algorithms.

[1165] Step 6:

[1166] The server organizes the classified data as structured data and stores it in a database. The data is stored in fields such as date, amount, and item name.

[1167] Step 7:

[1168] A user enters a search query through an application, for example, "electricity bill details for the last 3 months."

[1169] Step 8:

[1170] The terminal sends the user's search query to the server.

[1171] Step 9:

[1172] The server searches the database for data corresponding to the search query, using indexes to efficiently extract the data.

[1173] Step 10:

[1174] The server creates a summary of the search results using a summary generation algorithm as needed.

[1175] Step 11:

[1176] The server transmits the generated search results and summaries to the terminal.

[1177] Step 12:

[1178] The terminal displays search results and summaries to the user, allowing the user to quickly access the information they need.

[1179] Step 13:

[1180] The server analyzes the user's past search history, frequency of use, access patterns, etc. Based on this, it applies an algorithm to recommend important information and actions.

[1181] Step 14:

[1182] The server recommends information that it deems important to the user, and this information is sent to the device as a notification.

[1183] Step 15:

[1184] The terminal displays a notification of the recommended information to the user and provides an interface that allows the user to view detailed information.

[1185] These are the processing steps for a system that digitizes paper documents and performs management, search, summarization, reporting, and information recommendation, thereby simplifying paper document management for users and enabling quick access to the information they need.

[1186] Example 1

[1187] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1188] In today's world, homes are overflowing with paper documents, making it difficult to manage them efficiently. Manually searching through documents to find the information you need takes time and effort. It's also difficult to identify useful information based on its importance and frequency of use. There's a need for a system that can solve these issues and enable users to digitize paper documents and efficiently manage, search, summarize, report, and recommend information.

[1189] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1190] In this invention, the server includes a means for capturing paper documents as digital images, a means for converting the captured image data into text information, a means for classifying the text information and converting it into structured data, a data storage means, a means for a user to input a search query, a means for searching a data recording device based on the search query and displaying the results, a means for summarizing data based on the user's search query, and a means for recommending information based on user behavior data. This allows users to efficiently digitize paper documents and easily access the information. Furthermore, the system enables users to manage their lives smartly and efficiently through search, summarization, reporting, and information recommendations.

[1191] "Paper documents" refers to printed or written forms of paper, including electricity bills, shopping receipts, medical statements, insurance documents, school notices, pension newsletters, etc.

[1192] A "digital image" refers to an image file generated by photographing or scanning a physical paper document with a device such as a smartphone or scanner.

[1193] "Capture means" refers to a device and application software for capturing a paper document as a digital image.

[1194] "Character recognition means" refers to a technology that analyzes character information in an image and converts it into character data, and generally uses optical character recognition (OCR) technology.

[1195] "Data classification means" refers to algorithms and techniques for classifying extracted textual information into different categories (e.g., utility bills, receipts, medical statements, etc.).

[1196] "Data storage means" refers to a database system for long-term storage of information after data classification, which appropriately structures and stores digital data.

[1197] "Search input means" refers to an interface through which a user inputs a query to search for specific information, and sends this to a server via a terminal.

[1198] "Search result display means" refers to a function for displaying to a user data retrieved based on a search query, optionally providing results in a summarized form.

[1199] "Summary generation means" refers to the technology and algorithms used to summarize search results and information contained in a database and present it to the user in an easy-to-understand manner.

[1200] "Information recommendation means" refers to technology for recommending appropriate information and actions based on a user's usage history and behavioral data.

[1201] The system of the present invention is designed to efficiently digitize paper documents in the home and perform management, retrieval, summarization, reporting, and information recommendation. The structure of this system and its technical implementation are described in detail below.

[1202] System configuration and usage

[1203] Paper document capture method

[1204] Users use digital devices such as smartphones or scanners to capture digital images of paper documents (e.g., utility bills, receipts, medical statements, insurance documents, etc.) and can upload these digital images to the system through a dedicated application.

[1205] Character recognition means (OCR)

[1206] The digital image uploaded by the device is sent to a server, which then uses OCR technology, such as Amazon Textract or Google Cloud Vision, to extract text from the image. The OCR process includes preprocessing steps such as noise reduction and deskew.

[1207] Data Classification Methods

[1208] The server analyzes the extracted text and classifies the data based on the document type, using machine learning algorithms such as support vector machines (SVM) and random forests, for example, into categories such as medical statements, receipts, and utility bills.

[1209] Data storage means

[1210] The server stores the classified data in a database. The data is structured as a pair of item names and values. The database uses a database management system such as MySQL or PostgreSQL.

[1211] Search input method

[1212] The user enters a search query into the application's search bar. For example, the user might enter a query such as "electricity bill details for the past three months." The device then sends this search query to the server.

[1213] Search result display method

[1214] The server searches the database based on the received search query and extracts the relevant data. The search results are summarized and displayed to the user. The search function uses technologies such as Lucene and Elasticsearch.

[1215] Information recommendation means

[1216] The server analyzes the information based on the user's usage history and importance, and recommends appropriate information. For example, it could provide a function to notify users when their insurance contract is due for renewal.

[1217] Specific operation example

[1218] Utility bill management

[1219] Example prompt: "Please upload a photo of your electricity bill taken with your smartphone."

[1220] Users take a photo of their electricity bill with their smartphone and upload it to a dedicated app.

[1221] The device sends the uploaded image to the server.

[1222] The server uses OCR technology to extract text information from the image and classify it into the "utility bills" category.

[1223] The server stores the data in a database.

[1224] Search for medical details

[1225] Example prompt: "Please search for medical expense details for the past year."

[1226] A user types "medical expenses past year" into the app's search bar.

[1227] The device sends a search query to the server.

[1228] The server searches the database, extracts and summarizes the relevant medical details, and displays them to the user.

[1229] Generate monthly reports from receipts

[1230] Example prompt: "What is the total cost of food this month?"

[1231] The user enters "Total food expenses this month" into the app.

[1232] The device sends a query to the server.

[1233] The server aggregates the data for the specified period and generates a report.

[1234] The server sends the report to the user.

[1235] Recommendation of insurance documents

[1236] Example prompt: "I would like to receive insurance policy renewal notices."

[1237] The server analyzes the user's past search history and identifies important insurance-related information.

[1238] The server notifies the user that the time for renewal of the insurance contract is approaching and recommends related information to the user.

[1239] The user checks the recommended information in the app and takes the necessary action.

[1240] The system allows users to efficiently digitize paper documents and access information more quickly and easily, while search result summaries and information recommendations help users manage their lives more efficiently.

[1241] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1242] Step 1:

[1243] A user captures a paper document as a digital image using a smartphone or scanner.

[1244] Input: Paper document

[1245] Output: Digital image

[1246] Specific actions: The user launches the smartphone's camera app and takes a photo of a document, or scans a document using a scanner. The user then imports the captured or scanned image into the app.

[1247] Step 2:

[1248] The device uploads the digital images taken or scanned to a server via a dedicated app.

[1249] Input: Digital image

[1250] Output: Digital image sent to server

[1251] Specific operation: The user clicks the "Upload" button on the dedicated app to send the photographed or scanned image data to the cloud server. The device displays the sending status to the user.

[1252] Step 3:

[1253] The server performs OCR processing on the received digital image to extract text information.

[1254] Input: Digital image

[1255] Output: Extracted text information

[1256] How it works: The server analyzes the received image data using OCR technology such as Amazon Textract or Google Cloud Vision. The OCR process includes noise removal and tilt correction, and the character information in the image is extracted as text data.

[1257] Step 4:

[1258] The server analyzes the extracted text information and categorizes it based on the document type.

[1259] Input: Extracted text information

[1260] Output: Categorized data

[1261] How it works: The server analyzes text information using machine learning algorithms such as SVM and Random Forest. For example, the server identifies the keyword "electricity bill" and classifies it into the "utility bill" category.

[1262] Step 5:

[1263] The server stores the classified data in a database.

[1264] Input: Categorical data

[1265] Output: Data stored in the database

[1266] How it works: The server uses a database management system such as MySQL or PostgreSQL to store data categorized into categories in the form of item name / value pairs. For example, data in the "Utility Expenses" category is recorded in the form of "Date," "Amount," "Category," etc.

[1267] Step 6:

[1268] A user enters a search query into the search bar of an application, and the terminal sends the search query to a server.

[1269] Input: search query

[1270] Output: The search query sent to the server

[1271] What happens: The user enters "electricity bill details for the last 3 months" into the application's search bar and clicks the search button. The device sends the search query to the server.

[1272] Step 7:

[1273] The server searches the database based on the received search query, extracts and summarizes the relevant data.

[1274] Input: search query

[1275] Output: Summarized search results

[1276] What it does: The server uses technologies such as Lucene or Elasticsearch to search the database, extracts relevant data, and optionally summarizes the search results using an automatic summarization algorithm.

[1277] Step 8:

[1278] The server sends the summarized search results to the terminal, which displays the results to the user.

[1279] Input: Summarized search results

[1280] Output: The result displayed to the user

[1281] Specific operation: The server sends search results to the device in JSON format, etc. The device displays the received results in an easy-to-read format within the application interface, and the user confirms the search results.

[1282] Step 9:

[1283] The server analyzes the information based on the frequency of use and importance of the user, and recommends appropriate information to the user.

[1284] Input: User behavior data and usage history

[1285] Output: Recommendation notification

[1286] What happens: The server uses machine learning algorithms to analyze user behavior data. For example, it identifies when an insurance contract is due for renewal and notifies the user. The user then checks the recommendations in the app and takes the necessary action.

[1287] (Application example 1)

[1288] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1289] In recent years, the number of users of online shopping sites has increased, increasing the need for efficient management of purchase history and order details. However, managing paper-based statements is time-consuming, and it is not easy to search, summarize, or recommend information. In particular, it is difficult to summarize past purchase history or accurately recommend the next purchase date or related products. For this reason, there is a need for a system that improves user convenience and enables efficient data management and information provision.

[1290] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1291] In this invention, the server includes means for capturing paper documents as digital images, character recognition means for converting the captured image data into character information, data classification means for classifying the character information and converting it into structured data, database means for saving the data converted by the data classification means, search input means for a user to input a search query, search result display means for searching the database based on the search query and displaying the results, and information recommendation means for processing the user's purchase history and recommending the next purchase date and related products. This allows the user to efficiently manage their purchase history and order details and quickly receive searches, summaries, and recommendations.

[1292] "Means for capturing paper documents as digital images" refers to a function that allows a user to photograph a paper document using a smartphone or scanner and capture the image as a digital image.

[1293] "Character recognition means for converting captured image data into character information" is a function that uses OCR technology to extract character information from digital images.

[1294] The "data classification means for classifying character information and converting it into structured data" is a function that analyzes extracted character information, classifies the data based on the type and content of the document, and converts it into structured data in the form of item name and value pairs.

[1295] The "database means for storing data converted by the data classification means" is a function for storing structured data in a database so that it can be accessed efficiently later.

[1296] The "search input means by which a user inputs a search query" refers to a function by which a user inputs a search query to search for specific information in a search bar of an application, for example.

[1297] The "search result display means for searching a database based on a search query and displaying the results" is a function for retrieving data corresponding to the search query from a database and displaying the data to the user.

[1298] The "information recommendation means for processing the user's purchase history and recommending the next purchase time and related products" is a function that analyzes the user's past purchase history and recommends the next appropriate purchase time and related products.

[1299] As an embodiment of the present invention, we will explain a system that efficiently manages purchase histories and order details on an online shopping site and performs searches, summarization, and recommendations. This system has the following configuration and functions.

[1300] System configuration

[1301] 1. Paper document capture method

[1302] Users use their smartphone camera or a dedicated scanner to take a picture of the purchase receipt from the online shopping site and import it as a digital image.

[1303] 2. Character recognition method (OCR)

[1304] The server receives the digital image sent by the user and extracts text information from the image using OCR technology (such as pytesseract), as well as preprocessing the image by grayscale conversion and noise reduction.

[1305] 3. Data Classification Methods

[1306] The server analyzes the extracted text and classifies the data based on the document type (food, home appliances, clothing, etc.) and content using machine learning algorithms.

[1307] 4. Database Means

[1308] The server stores the classified data in a database. The stored data is structured and managed in the form of pairs of item names and values.

[1309] 5. Search Input Methods

[1310] A user types a search query into the search bar of a smartphone application, for example, "amount spent on food in the last 3 months."

[1311] 6. Search result display method

[1312] The server searches the database based on the search query and displays the relevant data to the user, optionally generating and displaying a summary of the search results.

[1313] 7. Information recommendation means

[1314] The server processes the user's purchase history and recommends the next purchase time and related products, for example, notifying the user that it is almost time to buy detergent again.

[1315] System operation example

[1316] 1. Managing purchase details

[1317] Users take a photo of their purchase details with their smartphone and upload it to a dedicated app. The device sends the image to a server, which converts it into text using OCR technology. The server then categorizes the information and organizes it into categories such as "food" or "home appliances."

[1318] 2. Search for food purchase amounts

[1319] If a user wants to find out how much they spent on food over the past three months, they simply enter "food spending over the past three months" into the application. The device sends the search query to the server, which then searches the database for the relevant data, summarizes it, and displays it.

[1320] 3. Recommendation for next purchase

[1321] Users receive recommendations for their next purchase and related products based on their purchase history. For example, if detergent stocks are running low, the server will send a notification recommending the next purchase.

[1322] Prompt Sentence Examples

[1323] Example 1: Want to know how much you spent on food over the past three months?

[1324] Input: "Food purchases for the past 3 months"

[1325] Output: "I've spent ¥5000 on food in the past three months."

[1326] Example 2: Recommending your next detergent purchase

[1327] Input: "Next recommended time to purchase detergent"

[1328] Output: "Your detergent stock is running low, so we recommend you buy more next time."

[1329] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1330] Step 1:

[1331] The user takes a photo of the purchase receipt with their smartphone and uploads the image to a dedicated app.

[1332] Input: Image of purchase details taken with a smartphone

[1333] Output: Digital image file

[1334] Specific operation: Use the smartphone camera to take a photo of a paper purchase invoice and upload the image to the server from the application.

[1335] Step 2:

[1336] The server applies OCR technology to convert the received image data into text information.

[1337] Input: Digital images uploaded by users

[1338] Output: Extracted text information

[1339] Specific operation: The server uses an image processing library (e.g., OpenCV) to convert the image to grayscale and perform noise reduction, then performs OCR using pytesseract.

[1340] Step 3:

[1341] The server analyzes the character information extracted by OCR, classifies the data, and structures it.

[1342] Input: Character information extracted by OCR

[1343] Output: Classified structured data

[1344] Specific operation: The server uses a machine learning algorithm to analyze the text information, classify the data by category or item, such as "food" or "home appliances," and structure it in the form of item name / value pairs.

[1345] Step 4:

[1346] The server stores the structured data in a database.

[1347] Input: Categorized and structured data

[1348] Output: Data stored in the database

[1349] Specific operation: The server connects to a database management system (e.g., MySQL) and stores the classified data by item.

[1350] Step 5:

[1351] A user types a search query into the search bar of a smartphone app.

[1352] Input: A search query such as "food spending in the last 3 months"

[1353] Output: None (because the query is sent to the server)

[1354] What happens: A user enters a search query into the application's search bar, and the query is sent to the server.

[1355] Step 6:

[1356] The server searches the database based on the search query and retrieves the relevant data.

[1357] Input: Search query

[1358] Output: Search results

[1359] What happens: The server runs an SQL query against the database to retrieve the relevant data, for example, the food purchase history for the past three months.

[1360] Step 7:

[1361] The server summarizes the retrieved data and displays it to the user.

[1362] Input: Search results

[1363] Output: Search results displayed as a summary

[1364] Specific operation: The server aggregates and analyzes the acquired data and organizes it as a summary. For example, it calculates the total amount of food purchases for the past three months and displays it to the user.

[1365] Step 8:

[1366] The server analyzes the user's purchasing history and recommends the next purchase time and related products.

[1367] Input: User purchase history data

[1368] Output: Recommendation

[1369] How it works: The server uses a recommendation algorithm to analyze the user's purchase history and generate recommendations for the next appropriate purchase time and related products. For example, if detergent stock is low, a notification will be sent recommending the next purchase.

[1370] Prompt Sentence Examples

[1371] Example 1: Want to know how much you spent on food over the past three months?

[1372] Input: "Food purchases for the past 3 months"

[1373] Output: "I've spent ¥5000 on food in the past three months."

[1374] Example 2: Recommending your next detergent purchase

[1375] Input: "Next recommended time to purchase detergent"

[1376] Output: "Your detergent stock is running low, so we recommend you buy more next time."

[1377] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1378] As an embodiment of the present invention, we will explain a system that digitizes paper documents present in the home and performs management, search, summarization, reporting, and information recommendation, as well as a system that combines an emotion engine that recognizes user emotions. This system has the following configuration and functions.

[1379] System configuration

[1380] 1. Paper document capture method

[1381] Users use a smartphone or scanner to take a photo of a paper document (utility bill, shopping receipt, medical bill, insurance document, school notice, pension newsletter, etc.) and capture it as a digital image.

[1382] 2. Character recognition method (OCR)

[1383] The server receives the digital image sent by the user and uses OCR technology to extract text information from the image. Character recognition is performed after preprocessing such as noise removal and skew correction.

[1384] 3. Data Classification Methods

[1385] The server analyzes the extracted text and classifies the data based on the document type (utility bill, receipt, medical statement, etc.) using machine learning algorithms.

[1386] 4. Database Means

[1387] The server stores the classified data in a database. The stored data is structured and managed in the form of pairs of item names and values.

[1388] 5. Search Input Methods

[1389] A user enters a search query into the application's search bar, for example, "electricity bill for the last 3 months."

[1390] 6. Search result display method

[1391] The server searches the database based on the search query and displays the relevant data to the user, optionally generating and displaying a summary of the search results.

[1392] 7. Information recommendation means

[1393] The server analyzes the information based on the frequency and importance of the user's use and recommends appropriate information to the user, such as notifying them that their insurance contract is due for renewal.

[1394] 8. Emotion recognition means

[1395] The server uses technology to recognize the user's emotions, for example, by analyzing the user's emotional state from their input speed, context, facial recognition, etc.

[1396] 9. Emotion Engine

[1397] The server then recommends information and adjusts the summary of search results based on the user's recognized emotions. The emotion engine changes its behavior depending on the user's emotional state, for example, displaying simple but important information when the user is stressed, or providing detailed information when the user is relaxed.

[1398] System operation example

[1399] 1. Utility bill management

[1400] Users take a photo of their electricity bill with their smartphone and upload it to a dedicated app.

[1401] The device sends the uploaded image to the server.

[1402] The server uses OCR technology to extract text from the image and classify it into the "utility bills" category.

[1403] The server stores the data in a database so that it can be quickly accessed when a user searches.

[1404] 2. Search for medical details

[1405] If a user wants to check the details of medical expenses for the past year, they enter "medical expenses for the past year" into the application.

[1406] The terminal sends a search query to the server, which searches the database for the relevant medical details.

[1407] The server displays the search results and their summaries to the user.

[1408] 3. Create monthly reports from receipts

[1409] If a user wants to know the total cost of food for the month, he / she inputs the request "Total cost of food for this month" into the application.

[1410] The terminal sends a request to the server, and the server aggregates the data for the specified period.

[1411] The server generates a report of the aggregated results and provides it to the user.

[1412] 4. Recommendation of insurance documents

[1413] The server analyzes the user's past search history and identifies important insurance-related information.

[1414] The server notifies the user that the time for renewal of the insurance contract is approaching and recommends related information to the user.

[1415] The user checks the recommended information in the app and takes the necessary action.

[1416] 5. Emotion-based information presentation

[1417] The server recognizes the user's emotions, and if the user is feeling stressed, for example, important and concise information is displayed preferentially.

[1418] If you're relaxed, it will show you more information and additional options.

[1419] This allows for optimal information provision according to the user's emotional state.

[1420] Summary

[1421] This invention allows users to efficiently digitize paper documents and quickly access the information they hold. In particular, by incorporating search, summarization, reporting, and information recommendation functions, as well as taking into account the user's emotions, it is possible to provide more personalized information. This system is expected to improve users' quality of life by eliminating the hassle of managing paper documents and reducing the time and stress involved.

[1422] The processing flow will be explained below.

[1423] Step 1:

[1424] Users take a photo of a paper document using a smartphone or scanner and import it as a digital image using a dedicated application.

[1425] Step 2:

[1426] The device sends the captured image to the server in an optimized format, compressing the image and adjusting the resolution to optimize the transfer speed.

[1427] Step 3:

[1428] The server temporarily stores the received digital images, along with the image's file format and metadata.

[1429] Step 4:

[1430] The server extracts text information from images using OCR (Optical Character Recognition) technology. Specifically, after preprocessing the image (noise removal, tilt correction, binarization, etc.), it detects text areas and recognizes the characters from them.

[1431] Step 5:

[1432] The server analyzes the extracted text and classifies the data based on the type of document, such as into categories like "utility bill," "receipt," or "medical statement," using machine learning algorithms.

[1433] Step 6:

[1434] The server organizes the classified data as structured data and stores it in a database. The data is stored in fields such as date, amount, and item name.

[1435] Step 7:

[1436] A user enters a search query through an application, for example, "electricity bill details for the last 3 months."

[1437] Step 8:

[1438] The terminal sends the user's search query to the server.

[1439] Step 9:

[1440] The server searches the database for data corresponding to the search query, using indexes to efficiently extract the data.

[1441] Step 10:

[1442] The server creates a summary of the search results using a summary generation algorithm as needed.

[1443] Step 11:

[1444] The server transmits the generated search results and summaries to the terminal.

[1445] Step 12:

[1446] The terminal displays search results and summaries to the user, allowing the user to quickly access the information they need.

[1447] Step 13:

[1448] The server analyzes the user's past search history, frequency of use, access patterns, etc. Based on this, it applies an algorithm to recommend important information and actions.

[1449] Step 14:

[1450] The server recommends information that it deems important to the user, and this information is sent to the device as a notification.

[1451] Step 15:

[1452] The terminal displays a notification of the recommended information to the user and provides an interface that allows the user to view detailed information.

[1453] Step 16:

[1454] The server uses technology to recognize the user's emotions, for example, by analyzing the user's emotional state from their typing speed, context, facial recognition, etc.

[1455] Step 17:

[1456] The server recommends information and adjusts the summary of search results based on the user's recognized emotions. The emotion engine displays simple, essential information when the user is stressed, and provides detailed information when the user is relaxed.

[1457] Step 18:

[1458] The device displays the most appropriate information according to the user's emotional state, thereby realizing personalized information provision that matches the user's emotional state.

[1459] These are the processing steps of the system that combines the emotion engine. This system simplifies the management of paper documents for users, enables quick access to necessary information, and provides information based on emotions.

[1460] Example 2

[1461] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1462] In conventional paper document management systems, digitizing and searching documents is cumbersome, and the accuracy of summarization and information recommendations is low.In addition, information presentation does not take into account the user's emotional state, and the user experience is not optimized.

[1463] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1464] In this invention, the server includes means for capturing paper documents as digital images, character recognition means for converting the captured image data into character information, data classification means for classifying the character information and converting it into structured data, database means for saving the data converted by the data classification means, search input means for a user to input a search query, search result display means for searching the database based on the search query and displaying the results, and emotion recognition means for recognizing the user's emotion and adjusting the way information is presented. This allows users to efficiently digitize documents, easily search and acquire information, and receive information optimally suited to their emotional state.

[1465] "Means for capturing paper documents as digital images" refers to a device or software that allows a user to photograph a paper document using a smartphone or scanner and capture the image as a digital image.

[1466] "Character recognition means" refers to a technology for extracting character information from a digital image, specifically using OCR (optical character recognition) technology.

[1467] A "data classification means" is an algorithm or machine learning model that analyzes extracted text information and classifies it into specific categories.

[1468] "Database means" refers to a database system for structuring and storing classified data, including SQL and NoSQL databases.

[1469] A "search input means" is an application or system that provides an interface for a user to enter a search query in order to search for specific information.

[1470] The "search result display means" is an interface for displaying to the user the data acquired based on the search query, and displays it as text or graphs on the screen.

[1471] "Emotion recognition means" is a technology for analyzing the user's emotional state, and recognizes emotions using data such as input speed, context, and facial recognition.

[1472] MODE FOR CARRYING OUT THE INVENTION

[1473] The system of the present invention efficiently digitizes paper documents kept at home by users and manages, searches, summarizes, reports, and recommends information. It also includes an emotion engine that recognizes the user's emotional state and provides information accordingly. This system operates using the following hardware and software:

[1474] Hardware and Software Configuration

[1475] 1. Paper document capture method

[1476] Users use a smartphone or scanner to capture digital images of paper documents (such as electricity bills, shopping receipts, medical bills, etc.), and then upload these images to the device using a dedicated application.

[1477] 2. Character recognition means (OCR technology)

[1478] The server receives the digital image sent by the user and uses OCR (Optical Character Recognition) technology to extract text information from the image. Specifically, a library such as Tesseract OCR is used to perform preprocessing such as noise removal and tilt correction before recognizing the text.

[1479] 3. Data Classification Methods

[1480] The server analyzes the text extracted by OCR and determines the document type using machine learning algorithms such as SVM (support vector machine) and neural networks. For example, an electricity bill would be classified as a "utility bill" category.

[1481] 4. Database Means

[1482] The server stores the classified data in a database using a structured database such as MySQL or PostgreSQL, where the data is stored in the form of field name / value pairs, allowing for fast and efficient data searches and report generation.

[1483] 5. Search Input Methods

[1484] The user enters a search query into the application's search bar, for example, a specific query such as "electricity bill details for the last three months." This search query is sent to the server via the device.

[1485] 6. Search result display method

[1486] The server searches the database based on the received search query and extracts the relevant data. It then generates a summary from the extracted data and displays the results on the user's device. For example, a search result for "electricity bill details for the past three months" would display the details for each month and the total amount.

[1487] 7. Information recommendation means

[1488] The server recommends appropriate information based on the user's usage history and importance. For example, if an insurance contract is due for renewal, the server notifies the user of this information and recommends detailed information. This ensures that the user does not miss important information.

[1489] 8. Emotion recognition means

[1490] The server analyzes the user's emotional state based on their input speed, context, facial recognition, etc. For example, it can accurately recognize the user's emotional state by using Python's OpenCV library and various cloud APIs (e.g., Azure's Emotion API). If the input speed is slow or if words indicating stress are detected in the context, it is determined that the user is feeling stressed.

[1491] 9. Emotion Engine

[1492] The server adjusts the way information is presented based on the user's recognized emotions. For example, if the user is stressed, only simple and important information is displayed, while if the user is relaxed, detailed information and additional options are provided. This allows the server to provide optimal information according to the user's emotional state.

[1493] Examples of concrete examples and prompts

[1494] Example 1: Utility bill management

[1495] Users take a photo of their electricity bill with their smartphone and upload it to a dedicated app.

[1496] The device sends the uploaded image to the server.

[1497] The server uses OCR technology to extract text from the image and classify it into the "utility bills" category.

[1498] The server stores the data in a database so that it can be quickly accessed when a user searches.

[1499] Example 2: Searching for medical details

[1500] If a user wants to check the details of medical expenses for the past year, they enter the search query "medical expenses for the past year."

[1501] The terminal sends a search query to the server, which searches the database for the relevant medical details.

[1502] The server displays the search results and their summaries to the user.

[1503] Example 3: Creating a monthly report from receipts

[1504] If the user wants to know the total food expenses for the month, he / she inputs a request such as "Total food expenses for this month."

[1505] The terminal sends a request to the server, and the server aggregates the data for the specified period.

[1506] The server generates a report of the aggregated results and provides it to the user.

[1507] Example 4: Recommending insurance documents

[1508] The server analyzes the user's past search history and identifies important insurance-related information.

[1509] The server notifies the user that the time for renewal of the insurance contract is approaching and recommends related information to the user.

[1510] The user reviews the recommended information and takes the necessary action.

[1511] Example 5: Emotion-based information provision

[1512] The server recognizes the user's emotions, and if the user is feeling stressed, for example, it prioritizes displaying concise and important information.

[1513] If the user is relaxed, provide more information or additional options, which provides optimal information to the user.

[1514] Example prompts to input to the generative AI model

[1515] Please tell me your electricity bill details for the past three months.

[1516] "Calculate the total cost of food last month."

[1517] Please show me your medical expenses for the past year.

[1518] "Can you tell me when my insurance policy is due for renewal?"

[1519] "Show me recommendations based on my current mood."

[1520] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1521] Step 1:

[1522] The user takes a photo of the document and captures it as a digital image.

[1523] A user takes a photo of a paper document using a smartphone or scanner. For example, they take a photo of an electricity bill. Then, they use a dedicated application to upload the captured image file to their device. The input is a paper document, and the output is a digital image file.

[1524] Step 2:

[1525] The device sends the digital image to the server

[1526] The terminal transfers the digital image file uploaded by the user to the server. The input is the digital image file, and the output is the digital image file stored on the server. This completes the communication until the image reaches the server.

[1527] Step 3:

[1528] The server extracts text information using OCR technology.

[1529] The server analyzes the received digital image and extracts text information from the image using OCR technology. Specifically, it uses the Tesseract OCR library and performs preprocessing such as noise removal and tilt correction. The input is the digital image file, and the output is the extracted text information.

[1530] Step 4:

[1531] The server classifies the data

[1532] The server analyzes the text extracted by OCR and uses machine learning algorithms (such as SVM or neural networks) to determine the type of document and classify it. For example, an electricity bill is classified into the "utility bill" category. The input is text, and the output is classified data.

[1533] Step 5:

[1534] The server stores the classified data in a database.

[1535] The server stores the classified data in a database. Specifically, it uses a MySQL or PostgreSQL database and stores the data in a structured format of field name / value pairs. The input is the classified data, and the output is the data stored in the database.

[1536] Step 6:

[1537] The user enters a search query

[1538] A user enters a query into the application's search bar to search for specific information. For example, a specific search query such as "electricity bill details for the last three months." The input is the user's search query, and the output is the transmission of the search query from the device to the server.

[1539] Step 7:

[1540] The server searches the database and displays the results

[1541] The server searches the database based on the received search query and extracts the relevant data. A summary is generated from the extracted data and the results are displayed on the user's device. For example, a search result for "electricity bill details for the past three months" would display the details for each month and the total amount. The input is the search query, and the output is the display of the search results.

[1542] Step 8:

[1543] Server recommends information

[1544] The server recommends appropriate information based on the user's usage history and importance. For example, if an insurance contract is due for renewal, the server notifies the user of this information and recommends detailed information. This prevents the user from missing important information. The input is the user's usage history and importance, and the output is a notification of recommended information.

[1545] Step 9:

[1546] The server recognizes emotions and adjusts information

[1547] The server analyzes and recognizes the user's emotional state based on their input speed, context, and facial recognition. Specifically, it uses the Python OpenCV library and Azure's Emotion API. For example, if the input speed is slow or if the user uses many words indicating stress, it determines that the user is feeling stressed. The input is the user's input data and video data, and the output is the analysis result of the user's emotional state.

[1548] Step 10:

[1549] The server provides information using an emotion engine

[1550] The server adjusts the way information is presented based on the user's recognized emotions. For example, if the user is stressed, it displays only simple and important information, while if the user is relaxed, it provides detailed information and additional options. The input is the analysis result of the user's emotional state, and the output is the adjusted information provided.

[1551] (Application example 2)

[1552] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1553] In conventional paper document management systems, managing paper coupons and receipts is cumbersome, making it difficult to digitize them for easy user use. Furthermore, providing information according to the user's emotional state is not considered, which does not lead to an improved user experience. This has led to a demand for a system that allows users to efficiently manage paper information and quickly obtain the information they need without feeling stressed.

[1554] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1555] In this invention, the server includes means for capturing paper documents as digital images, character recognition means for converting the captured image data into character information, data classification means for classifying the character information and converting it into structured data, database means for saving the data converted by the data classification means, search input means for a user to input a search query, search result display means for searching the database based on the search query and displaying the results, emotion recognition means for recognizing the emotional state of the user, and emotion engine means for adjusting the method of providing information based on the recognized emotional state. This enables users to efficiently digitize and manage paper coupons and receipts, and further enables providing information according to the user's emotional state, thereby providing a more comfortable and personalized user experience.

[1556] - "Means for capturing paper documents" means means for capturing paper documents as digital images.

[1557] The "character recognition means" is a means for converting captured digital image data into character information.

[1558] The "data classification means" is a means for classifying the character information converted by the character recognition means into categories and converting it into structured data.

[1559] "Database Means" means a means for storing classified structured data and making it accessible as needed.

[1560] A "search input means" is a means by which a user inputs a search query.

[1561] The "search result display means" is a means for searching a database based on an input search query and displaying the results to the user.

[1562] The "emotion recognition means" is a means for analyzing and recognizing the user's emotional state.

[1563] An "emotion engine means" is a means for adjusting how information is presented based on the perceived emotional state of the user.

[1564] This invention describes a system for improving the shopping experience for brick-and-mortar stores that digitizes paper documents in the home and performs management, search, summarization, and information recommendation, in addition to a system that combines an emotion engine that recognizes user emotions.

[1565] System configuration and functions

[1566] This system has the following configuration and functions:

[1567] 1. Paper document capture method

[1568] This is a method for users to take a photo of a paper coupon or receipt using their smartphone camera and import it as a digital image. For example, a user can take a photo of a receipt they receive at a store cash register using a dedicated app.

[1569] 2. Character recognition method (OCR)

[1570] The server receives the digital image sent by the user and uses OCR technology to extract text information from the image. The Google Cloud Vision API is used to achieve highly accurate character recognition.

[1571] 3. Data Classification Methods

[1572] The server analyzes the extracted text and categorizes the data based on the type of coupon or receipt, using a machine learning algorithm to categorize the data into categories such as food, daily necessities, and services.

[1573] 4. Database Means

[1574] The server stores the classified data in the Firebase database and manages it as structured data, allowing users to quickly access the information they need.

[1575] 5. Search Input Methods

[1576] A way for a user to enter a search query into a search bar within an application, for example, "Show me this month's grocery receipts."

[1577] 6. Search result display method

[1578] It is a way for the server to search the Firebase database based on a search query and display the relevant data to the user, along with generating a summary of the search results.

[1579] 7. Information recommendation means

[1580] This is a means by which the server analyzes information based on the frequency and importance of the user's use and recommends appropriate coupons and products to the user. For example, it analyzes the user's purchase history and recommends coupons for products that the user is likely to purchase next.

[1581] 8. Emotion recognition means

[1582] The server uses technology to recognize the user's emotions. It uses Amazon Rekognition to analyze the user's emotional state from their typing speed and facial expressions.

[1583] 9. Emotional Engine Means

[1584] It is a way for the server to recommend information and tailor search result summaries based on the user's perceived emotions. The emotion engine can present simple, essential information when the user is stressed, and detailed information when the user is relaxed.

[1585] Specific examples

[1586] For example, if a user types "Show this month's grocery receipts" into a smartphone app, the following will happen:

[1587] 1. The user takes a photo of the receipt with their smartphone and uploads it to the app.

[1588] 2. The app sends the digital image to a server.

[1589] 3. The server uses the Google Cloud Vision API to extract text from the image.

[1590] 4. The server classifies the extracted text information into categories (food, daily necessities, etc.) and stores it in the Firebase database.

[1591] 5. The user uses a search input method to enter the search query "Show this month's grocery receipts."

[1592] 6. The server searches the database, provides the relevant receipt information in a search result display means, and generates a summary.

[1593] 7. Use Amazon Rekognition to analyze user sentiment and adjust how information is displayed using an emotion engine.

[1594] Prompt Sentence Examples

[1595] When the user enters "Show this month's food receipts," the server displays the corresponding receipt information, and can provide information according to the user's emotions.

[1596] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1597] Step 1:

[1598] The user takes a photo of a paper coupon or receipt with their smartphone.

[1599] Input: Paper coupons and receipts

[1600] Output: Coupons and receipts as digital images

[1601] How it works: The user takes a photo of a paper coupon or receipt using the camera function of the dedicated app. The captured image is saved to the smartphone's storage.

[1602] Step 2:

[1603] The terminal transmits the captured digital image to the server.

[1604] Input: The digital image acquired in step 1

[1605] Output: Digital image sent to server

[1606] How it works: The app sends the captured image to a cloud server, along with image metadata (such as the date and time of the photo and user information).

[1607] Step 3:

[1608] The server performs OCR processing to extract characters from the digital image.

[1609] Input: Digital image sent to the server

[1610] Output: Extracted text information

[1611] How it works: The server uses the Google Cloud Vision API to extract text from digital images, including preprocessing such as noise removal and deskewing.

[1612] Step 4:

[1613] The server classifies the extracted text information into categories.

[1614] Input: Character information extracted in step 3

[1615] Output: Classified text information

[1616] How it works: The server uses machine learning algorithms to classify the extracted text into categories such as coupons, receipts, food, and household items.

[1617] Step 5:

[1618] The server stores the classified character information in a database.

[1619] Input: Character information classified in step 4

[1620] Output: Text information stored in the database

[1621] How it works: The server stores classified textual information in the Firebase database. The stored data is managed as structured data, making it easy to search.

[1622] Step 6:

[1623] A user enters a search query into a search bar within the application.

[1624] Input: User's search query (e.g., "View this month's grocery receipts")

[1625] Output: The search query is sent to the server

[1626] What happens: A user types a query into an application's search bar, for example, "Show me this month's grocery receipts."

[1627] Step 7:

[1628] The server searches the database and retrieves the relevant data.

[1629] Input: The search query entered in step 6

[1630] Output: Data as search results

[1631] How it works: Your server searches your Firebase database to retrieve data that matches your query. For example, it aggregates relevant receipt information to display "Grocery Receipts of the Month."

[1632] Step 8:

[1633] The server generates and displays the search results and their summaries.

[1634] Input: Search results obtained in step 7

[1635] Output: Summarized search results

[1636] How it works: The server generates a summary based on the search results, for example aggregating multiple receipts and displaying the total amount and key purchase items.

[1637] Step 9:

[1638] The server recognizes the user's emotions and adjusts the display method according to that information.

[1639] Input: User's face image and input speed information

[1640] Output: How to display the adjusted information

[1641] How it works: The server uses Amazon Rekognition to analyze the user's facial image and typing speed to recognize their emotional state. For example, if the user is stressed, it will display simple, essential information, and if they are relaxed, it will provide detailed information.

[1642] By implementing the above steps, users can efficiently digitize and manage paper coupons and receipts, and quickly search and retrieve the information they need. Furthermore, the emotion engine makes it possible to provide personalized information based on the user's emotional state.

[1643] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1644] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1645] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1646] [Fourth embodiment]

[1647] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1648] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1649] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1650] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1651] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1652] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1653] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1654] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1655] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1656] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1657] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1658] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1659] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1660] As an embodiment of the present invention, a system for digitizing paper documents present in the home and managing, searching, summarizing, reporting, and recommending information will be described. This system has the following configuration and functions.

[1661] System configuration

[1662] 1. Paper document capture method

[1663] Users use a smartphone or scanner to take a photo of a paper document (utility bill, shopping receipt, medical bill, insurance document, school notice, pension newsletter, etc.) and capture it as a digital image.

[1664] 2. Character recognition method (OCR)

[1665] The server receives the digital image sent by the user and uses OCR technology to extract text information from the image. Character recognition is performed after preprocessing such as noise removal and skew correction.

[1666] 3. Data Classification Methods

[1667] The server analyzes the extracted text and classifies the data based on the document type (utility bill, receipt, medical statement, etc.) using machine learning algorithms.

[1668] 4. Database Means

[1669] The server stores the classified data in a database. The stored data is structured and managed in the form of pairs of item names and values.

[1670] 5. Search Input Methods

[1671] A user enters a search query into the application's search bar, for example, "electricity bill for the last 3 months."

[1672] 6. Search result display method

[1673] The server searches the database based on the search query and displays the relevant data to the user, optionally generating and displaying a summary of the search results.

[1674] 7. Information recommendation means

[1675] The server analyzes the information based on the frequency and importance of the user's use and recommends appropriate information to the user, such as notifying them that their insurance contract is due for renewal.

[1676] System operation example

[1677] 1. Utility bill management

[1678] Users take a photo of their electricity bill with their smartphone and upload it to a dedicated app.

[1679] The device sends the uploaded image to the server.

[1680] The server uses OCR technology to extract text from the image and classify it into the "utility bills" category.

[1681] The server stores the data in a database so that it can be quickly accessed when a user searches.

[1682] 2. Search for medical details

[1683] If a user wants to check the details of medical expenses for the past year, they enter "medical expenses for the past year" into the application.

[1684] The terminal sends a search query to the server, which searches the database for the relevant medical details.

[1685] The server displays the search results and their summaries to the user.

[1686] 3. Create monthly reports from receipts

[1687] If a user wants to know the total cost of food for the month, he / she inputs the request "Total cost of food for this month" into the application.

[1688] The terminal sends a request to the server, and the server aggregates the data for the specified period.

[1689] The server generates a report of the aggregated results and provides it to the user.

[1690] 4. Recommendation of insurance documents

[1691] The server analyzes the user's past search history and identifies important insurance-related information.

[1692] The server notifies the user that the time for renewal of the insurance contract is approaching and recommends related information to the user.

[1693] The user checks the recommended information in the app and takes the necessary action.

[1694] Summary

[1695] This invention allows users to efficiently digitize paper documents and quickly access the information contained therein. In particular, it allows users to manage their lives smartly and efficiently through search, summarization, reporting, and information recommendation functions. This system is expected to improve the quality of life by eliminating the complexity of managing paper documents for users and reducing time and stress.

[1696] The processing flow will be explained below.

[1697] Step 1:

[1698] Users take a photo of a paper document using a smartphone or scanner and import it as a digital image using a dedicated application.

[1699] Step 2:

[1700] The device sends the captured image to the server in an optimized format, compressing the image and adjusting the resolution to optimize the transfer speed.

[1701] Step 3:

[1702] The server temporarily stores the received digital images, along with the image's file format and metadata.

[1703] Step 4:

[1704] The server extracts text information from images using OCR (Optical Character Recognition) technology. Specifically, after preprocessing the image (noise removal, tilt correction, binarization, etc.), it detects text areas and recognizes the characters from them.

[1705] Step 5:

[1706] The server analyzes the extracted text and classifies the data based on the type of document, such as into categories like "utility bill," "receipt," or "medical statement," using machine learning algorithms.

[1707] Step 6:

[1708] The server organizes the classified data as structured data and stores it in a database. The data is stored in fields such as date, amount, and item name.

[1709] Step 7:

[1710] A user enters a search query through an application, for example, "electricity bill details for the last 3 months."

[1711] Step 8:

[1712] The terminal sends the user's search query to the server.

[1713] Step 9:

[1714] The server searches the database for data corresponding to the search query, using indexes to efficiently extract the data.

[1715] Step 10:

[1716] The server creates a summary of the search results using a summary generation algorithm as needed.

[1717] Step 11:

[1718] The server transmits the generated search results and summaries to the terminal.

[1719] Step 12:

[1720] The terminal displays search results and summaries to the user, allowing the user to quickly access the information they need.

[1721] Step 13:

[1722] The server analyzes the user's past search history, frequency of use, access patterns, etc. Based on this, it applies an algorithm to recommend important information and actions.

[1723] Step 14:

[1724] The server recommends information that it deems important to the user, and this information is sent to the device as a notification.

[1725] Step 15:

[1726] The terminal displays a notification of the recommended information to the user and provides an interface that allows the user to view detailed information.

[1727] These are the processing steps for a system that digitizes paper documents and performs management, search, summarization, reporting, and information recommendation, thereby simplifying paper document management for users and enabling quick access to the information they need.

[1728] Example 1

[1729] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1730] In today's world, homes are overflowing with paper documents, making it difficult to manage them efficiently. Manually searching through documents to find the information you need takes time and effort. It's also difficult to identify useful information based on its importance and frequency of use. There's a need for a system that can solve these issues and enable users to digitize paper documents and efficiently manage, search, summarize, report, and recommend information.

[1731] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1732] In this invention, the server includes a means for capturing paper documents as digital images, a means for converting the captured image data into text information, a means for classifying the text information and converting it into structured data, a data storage means, a means for a user to input a search query, a means for searching a data recording device based on the search query and displaying the results, a means for summarizing data based on the user's search query, and a means for recommending information based on user behavior data. This allows users to efficiently digitize paper documents and easily access the information. Furthermore, the system enables users to manage their lives smartly and efficiently through search, summarization, reporting, and information recommendations.

[1733] "Paper documents" refers to printed or written forms of paper, including electricity bills, shopping receipts, medical statements, insurance documents, school notices, pension newsletters, etc.

[1734] A "digital image" refers to an image file generated by photographing or scanning a physical paper document with a device such as a smartphone or scanner.

[1735] "Capture means" refers to a device and application software for capturing a paper document as a digital image.

[1736] "Character recognition means" refers to a technology that analyzes character information in an image and converts it into character data, and generally uses optical character recognition (OCR) technology.

[1737] "Data classification means" refers to algorithms and techniques for classifying extracted textual information into different categories (e.g., utility bills, receipts, medical statements, etc.).

[1738] "Data storage means" refers to a database system for long-term storage of information after data classification, which appropriately structures and stores digital data.

[1739] "Search input means" refers to an interface through which a user inputs a query to search for specific information, and sends this to a server via a terminal.

[1740] "Search result display means" refers to a function for displaying to a user data retrieved based on a search query, optionally providing results in a summarized form.

[1741] "Summary generation means" refers to the technology and algorithms used to summarize search results and information contained in a database and present it to the user in an easy-to-understand manner.

[1742] "Information recommendation means" refers to technology for recommending appropriate information and actions based on a user's usage history and behavioral data.

[1743] The system of the present invention is designed to efficiently digitize paper documents in the home and perform management, retrieval, summarization, reporting, and information recommendation. The structure of this system and its technical implementation are described in detail below.

[1744] System configuration and usage

[1745] Paper document capture method

[1746] Users use digital devices such as smartphones or scanners to capture digital images of paper documents (e.g., utility bills, receipts, medical statements, insurance documents, etc.) and can upload these digital images to the system through a dedicated application.

[1747] Character recognition means (OCR)

[1748] The digital image uploaded by the device is sent to a server, which then uses OCR technology, such as Amazon Textract or Google Cloud Vision, to extract text from the image. The OCR process includes preprocessing steps such as noise reduction and deskew.

[1749] Data Classification Methods

[1750] The server analyzes the extracted text and classifies the data based on the document type, using machine learning algorithms such as support vector machines (SVM) and random forests, for example, into categories such as medical statements, receipts, and utility bills.

[1751] Data storage means

[1752] The server stores the classified data in a database. The data is structured as a pair of item names and values. The database uses a database management system such as MySQL or PostgreSQL.

[1753] Search input method

[1754] The user enters a search query into the application's search bar. For example, the user might enter a query such as "electricity bill details for the past three months." The device then sends this search query to the server.

[1755] Search result display method

[1756] The server searches the database based on the received search query and extracts the relevant data. The search results are summarized and displayed to the user. The search function uses technologies such as Lucene and Elasticsearch.

[1757] Information recommendation means

[1758] The server analyzes the information based on the user's usage history and importance, and recommends appropriate information. For example, it could provide a function to notify users when their insurance contract is due for renewal.

[1759] Specific operation example

[1760] Utility bill management

[1761] Example prompt: "Please upload a photo of your electricity bill taken with your smartphone."

[1762] Users take a photo of their electricity bill with their smartphone and upload it to a dedicated app.

[1763] The device sends the uploaded image to the server.

[1764] The server uses OCR technology to extract text information from the image and classify it into the "utility bills" category.

[1765] The server stores the data in a database.

[1766] Search for medical details

[1767] Example prompt: "Please search for medical expense details for the past year."

[1768] A user types "medical expenses past year" into the app's search bar.

[1769] The device sends a search query to the server.

[1770] The server searches the database, extracts and summarizes the relevant medical details, and displays them to the user.

[1771] Generate monthly reports from receipts

[1772] Example prompt: "What is the total cost of food this month?"

[1773] The user enters "Total food expenses this month" into the app.

[1774] The device sends a query to the server.

[1775] The server aggregates the data for the specified period and generates a report.

[1776] The server sends the report to the user.

[1777] Recommendation of insurance documents

[1778] Example prompt: "I would like to receive insurance policy renewal notices."

[1779] The server analyzes the user's past search history and identifies important insurance-related information.

[1780] The server notifies the user that the time for renewal of the insurance contract is approaching and recommends related information to the user.

[1781] The user checks the recommended information in the app and takes the necessary action.

[1782] The system allows users to efficiently digitize paper documents and access information more quickly and easily, while search result summaries and information recommendations help users manage their lives more efficiently.

[1783] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1784] Step 1:

[1785] A user captures a paper document as a digital image using a smartphone or scanner.

[1786] Input: Paper document

[1787] Output: Digital image

[1788] Specific actions: The user launches the smartphone's camera app and takes a photo of a document, or scans a document using a scanner. The user then imports the captured or scanned image into the app.

[1789] Step 2:

[1790] The device uploads the digital images taken or scanned to a server via a dedicated app.

[1791] Input: Digital image

[1792] Output: Digital image sent to server

[1793] Specific operation: The user clicks the "Upload" button on the dedicated app to send the photographed or scanned image data to the cloud server. The device displays the sending status to the user.

[1794] Step 3:

[1795] The server performs OCR processing on the received digital image to extract text information.

[1796] Input: Digital image

[1797] Output: Extracted text information

[1798] How it works: The server analyzes the received image data using OCR technology such as Amazon Textract or Google Cloud Vision. The OCR process includes noise removal and tilt correction, and the character information in the image is extracted as text data.

[1799] Step 4:

[1800] The server analyzes the extracted text information and categorizes it based on the document type.

[1801] Input: Extracted text information

[1802] Output: Categorized data

[1803] How it works: The server analyzes text information using machine learning algorithms such as SVM and Random Forest. For example, the server identifies the keyword "electricity bill" and classifies it into the "utility bill" category.

[1804] Step 5:

[1805] The server stores the classified data in a database.

[1806] Input: Categorical data

[1807] Output: Data stored in the database

[1808] How it works: The server uses a database management system such as MySQL or PostgreSQL to store data categorized into categories in the form of item name / value pairs. For example, data in the "Utility Expenses" category is recorded in the form of "Date," "Amount," "Category," etc.

[1809] Step 6:

[1810] A user enters a search query into the search bar of an application, and the terminal sends the search query to a server.

[1811] Input: search query

[1812] Output: The search query sent to the server

[1813] What happens: The user enters "electricity bill details for the last 3 months" into the application's search bar and clicks the search button. The device sends the search query to the server.

[1814] Step 7:

[1815] The server searches the database based on the received search query, extracts and summarizes the relevant data.

[1816] Input: search query

[1817] Output: Summarized search results

[1818] What it does: The server uses technologies such as Lucene or Elasticsearch to search the database, extracts relevant data, and optionally summarizes the search results using an automatic summarization algorithm.

[1819] Step 8:

[1820] The server sends the summarized search results to the terminal, which displays the results to the user.

[1821] Input: Summarized search results

[1822] Output: The result displayed to the user

[1823] Specific operation: The server sends search results to the device in JSON format, etc. The device displays the received results in an easy-to-read format within the application interface, and the user confirms the search results.

[1824] Step 9:

[1825] The server analyzes the information based on the frequency of use and importance of the user, and recommends appropriate information to the user.

[1826] Input: User behavior data and usage history

[1827] Output: Recommendation notification

[1828] What happens: The server uses machine learning algorithms to analyze user behavior data. For example, it identifies when an insurance contract is due for renewal and notifies the user. The user then checks the recommendations in the app and takes the necessary action.

[1829] (Application example 1)

[1830] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1831] In recent years, the number of users of online shopping sites has increased, increasing the need for efficient management of purchase history and order details. However, managing paper-based statements is time-consuming, and it is not easy to search, summarize, or recommend information. In particular, it is difficult to summarize past purchase history or accurately recommend the next purchase date or related products. For this reason, there is a need for a system that improves user convenience and enables efficient data management and information provision.

[1832] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1833] In this invention, the server includes means for capturing paper documents as digital images, character recognition means for converting the captured image data into character information, data classification means for classifying the character information and converting it into structured data, database means for saving the data converted by the data classification means, search input means for a user to input a search query, search result display means for searching the database based on the search query and displaying the results, and information recommendation means for processing the user's purchase history and recommending the next purchase date and related products. This allows the user to efficiently manage their purchase history and order details and quickly receive searches, summaries, and recommendations.

[1834] "Means for capturing paper documents as digital images" refers to a function that allows a user to photograph a paper document using a smartphone or scanner and capture the image as a digital image.

[1835] "Character recognition means for converting captured image data into character information" is a function that uses OCR technology to extract character information from digital images.

[1836] The "data classification means for classifying character information and converting it into structured data" is a function that analyzes extracted character information, classifies the data based on the type and content of the document, and converts it into structured data in the form of item name and value pairs.

[1837] The "database means for storing data converted by the data classification means" is a function for storing structured data in a database so that it can be accessed efficiently later.

[1838] The "search input means by which a user inputs a search query" refers to a function by which a user inputs a search query to search for specific information in a search bar of an application, for example.

[1839] The "search result display means for searching a database based on a search query and displaying the results" is a function for retrieving data corresponding to the search query from a database and displaying the data to the user.

[1840] The "information recommendation means for processing the user's purchase history and recommending the next purchase time and related products" is a function that analyzes the user's past purchase history and recommends the next appropriate purchase time and related products.

[1841] As an embodiment of the present invention, we will explain a system that efficiently manages purchase histories and order details on an online shopping site and performs searches, summarization, and recommendations. This system has the following configuration and functions.

[1842] System configuration

[1843] 1. Paper document capture method

[1844] Users use their smartphone camera or a dedicated scanner to take a picture of the purchase receipt from the online shopping site and import it as a digital image.

[1845] 2. Character recognition method (OCR)

[1846] The server receives the digital image sent by the user and extracts text information from the image using OCR technology (such as pytesseract), as well as preprocessing the image by grayscale conversion and noise reduction.

[1847] 3. Data Classification Methods

[1848] The server analyzes the extracted text and classifies the data based on the document type (food, home appliances, clothing, etc.) and content using machine learning algorithms.

[1849] 4. Database Means

[1850] The server stores the classified data in a database. The stored data is structured and managed in the form of pairs of item names and values.

[1851] 5. Search Input Methods

[1852] A user types a search query into the search bar of a smartphone application, for example, "amount spent on food in the last 3 months."

[1853] 6. Search result display method

[1854] The server searches the database based on the search query and displays the relevant data to the user, optionally generating and displaying a summary of the search results.

[1855] 7. Information recommendation means

[1856] The server processes the user's purchase history and recommends the next purchase time and related products, for example, notifying the user that it is almost time to buy detergent again.

[1857] System operation example

[1858] 1. Managing purchase details

[1859] Users take a photo of their purchase details with their smartphone and upload it to a dedicated app. The device sends the image to a server, which converts it into text using OCR technology. The server then categorizes the information and organizes it into categories such as "food" or "home appliances."

[1860] 2. Search for food purchase amounts

[1861] If a user wants to find out how much they spent on food over the past three months, they simply enter "food spending over the past three months" into the application. The device sends the search query to the server, which then searches the database for the relevant data, summarizes it, and displays it.

[1862] 3. Recommendation for next purchase

[1863] Users receive recommendations for their next purchase and related products based on their purchase history. For example, if detergent stocks are running low, the server will send a notification recommending the next purchase.

[1864] Prompt Sentence Examples

[1865] Example 1: Want to know how much you spent on food over the past three months?

[1866] Input: "Food purchases for the past 3 months"

[1867] Output: "I've spent ¥5000 on food in the past three months."

[1868] Example 2: Recommending your next detergent purchase

[1869] Input: "Next recommended time to purchase detergent"

[1870] Output: "Your detergent stock is running low, so we recommend you buy more next time."

[1871] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1872] Step 1:

[1873] The user takes a photo of the purchase receipt with their smartphone and uploads the image to a dedicated app.

[1874] Input: Image of purchase details taken with a smartphone

[1875] Output: Digital image file

[1876] Specific operation: Use the smartphone camera to take a photo of a paper purchase invoice and upload the image to the server from the application.

[1877] Step 2:

[1878] The server applies OCR technology to convert the received image data into text information.

[1879] Input: Digital images uploaded by users

[1880] Output: Extracted text information

[1881] Specific operation: The server uses an image processing library (e.g., OpenCV) to convert the image to grayscale and perform noise reduction, then performs OCR using pytesseract.

[1882] Step 3:

[1883] The server analyzes the character information extracted by OCR, classifies the data, and structures it.

[1884] Input: Character information extracted by OCR

[1885] Output: Classified structured data

[1886] Specific operation: The server uses a machine learning algorithm to analyze the text information, classify the data by category or item, such as "food" or "home appliances," and structure it in the form of item name / value pairs.

[1887] Step 4:

[1888] The server stores the structured data in a database.

[1889] Input: Categorized and structured data

[1890] Output: Data stored in the database

[1891] Specific operation: The server connects to a database management system (e.g., MySQL) and stores the classified data by item.

[1892] Step 5:

[1893] A user types a search query into the search bar of a smartphone app.

[1894] Input: A search query such as "food spending in the last 3 months"

[1895] Output: None (because the query is sent to the server)

[1896] What happens: A user enters a search query into the application's search bar, and the query is sent to the server.

[1897] Step 6:

[1898] The server searches the database based on the search query and retrieves the relevant data.

[1899] Input: Search query

[1900] Output: Search results

[1901] What happens: The server runs an SQL query against the database to retrieve the relevant data, for example, the food purchase history for the past three months.

[1902] Step 7:

[1903] The server summarizes the retrieved data and displays it to the user.

[1904] Input: Search results

[1905] Output: Search results displayed as a summary

[1906] Specific operation: The server aggregates and analyzes the acquired data and organizes it as a summary. For example, it calculates the total amount of food purchases for the past three months and displays it to the user.

[1907] Step 8:

[1908] The server analyzes the user's purchasing history and recommends the next purchase time and related products.

[1909] Input: User purchase history data

[1910] Output: Recommendation

[1911] How it works: The server uses a recommendation algorithm to analyze the user's purchase history and generate recommendations for the next appropriate purchase time and related products. For example, if detergent stock is low, a notification will be sent recommending the next purchase.

[1912] Prompt Sentence Examples

[1913] Example 1: Want to know how much you spent on food over the past three months?

[1914] Input: "Food purchases for the past 3 months"

[1915] Output: "I've spent ¥5000 on food in the past three months."

[1916] Example 2: Recommending your next detergent purchase

[1917] Input: "Next recommended time to purchase detergent"

[1918] Output: "Your detergent stock is running low, so we recommend you buy more next time."

[1919] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1920] As an embodiment of the present invention, we will explain a system that digitizes paper documents present in the home and performs management, search, summarization, reporting, and information recommendation, as well as a system that combines an emotion engine that recognizes user emotions. This system has the following configuration and functions.

[1921] System configuration

[1922] 1. Paper document capture method

[1923] Users use a smartphone or scanner to take a photo of a paper document (utility bill, shopping receipt, medical bill, insurance document, school notice, pension newsletter, etc.) and capture it as a digital image.

[1924] 2. Character recognition method (OCR)

[1925] The server receives the digital image sent by the user and uses OCR technology to extract text information from the image. Character recognition is performed after preprocessing such as noise removal and skew correction.

[1926] 3. Data Classification Methods

[1927] The server analyzes the extracted text and classifies the data based on the document type (utility bill, receipt, medical statement, etc.) using machine learning algorithms.

[1928] 4. Database Means

[1929] The server stores the classified data in a database. The stored data is structured and managed in the form of pairs of item names and values.

[1930] 5. Search Input Methods

[1931] A user enters a search query into the application's search bar, for example, "electricity bill for the last 3 months."

[1932] 6. Search result display method

[1933] The server searches the database based on the search query and displays the relevant data to the user, optionally generating and displaying a summary of the search results.

[1934] 7. Information recommendation means

[1935] The server analyzes the information based on the frequency and importance of the user's use and recommends appropriate information to the user, such as notifying them that their insurance contract is due for renewal.

[1936] 8. Emotion recognition means

[1937] The server uses technology to recognize the user's emotions, for example, by analyzing the user's emotional state from their input speed, context, facial recognition, etc.

[1938] 9. Emotion Engine

[1939] The server then recommends information and adjusts the summary of search results based on the user's recognized emotions. The emotion engine changes its behavior depending on the user's emotional state, for example, displaying simple but important information when the user is stressed, or providing detailed information when the user is relaxed.

[1940] System operation example

[1941] 1. Utility bill management

[1942] Users take a photo of their electricity bill with their smartphone and upload it to a dedicated app.

[1943] The device sends the uploaded image to the server.

[1944] The server uses OCR technology to extract text from the image and classify it into the "utility bills" category.

[1945] The server stores the data in a database so that it can be quickly accessed when a user searches.

[1946] 2. Search for medical details

[1947] If a user wants to check the details of medical expenses for the past year, they enter "medical expenses for the past year" into the application.

[1948] The terminal sends a search query to the server, which searches the database for the relevant medical details.

[1949] The server displays the search results and their summaries to the user.

[1950] 3. Create monthly reports from receipts

[1951] If a user wants to know the total cost of food for the month, he / she inputs the request "Total cost of food for this month" into the application.

[1952] The terminal sends a request to the server, and the server aggregates the data for the specified period.

[1953] The server generates a report of the aggregated results and provides it to the user.

[1954] 4. Recommendation of insurance documents

[1955] The server analyzes the user's past search history and identifies important insurance-related information.

[1956] The server notifies the user that the time for renewal of the insurance contract is approaching and recommends related information to the user.

[1957] The user checks the recommended information in the app and takes the necessary action.

[1958] 5. Emotion-based information presentation

[1959] The server recognizes the user's emotions, and if the user is feeling stressed, for example, important and concise information is displayed preferentially.

[1960] If you're relaxed, it will show you more information and additional options.

[1961] This allows for optimal information provision according to the user's emotional state.

[1962] Summary

[1963] This invention allows users to efficiently digitize paper documents and quickly access the information they hold. In particular, by incorporating search, summarization, reporting, and information recommendation functions, as well as taking into account the user's emotions, it is possible to provide more personalized information. This system is expected to improve users' quality of life by eliminating the hassle of managing paper documents and reducing the time and stress involved.

[1964] The processing flow will be explained below.

[1965] Step 1:

[1966] Users take a photo of a paper document using a smartphone or scanner and import it as a digital image using a dedicated application.

[1967] Step 2:

[1968] The device sends the captured image to the server in an optimized format, compressing the image and adjusting the resolution to optimize the transfer speed.

[1969] Step 3:

[1970] The server temporarily stores the received digital images, along with the image's file format and metadata.

[1971] Step 4:

[1972] The server extracts text information from images using OCR (Optical Character Recognition) technology. Specifically, after preprocessing the image (noise removal, tilt correction, binarization, etc.), it detects text areas and recognizes the characters from them.

[1973] Step 5:

[1974] The server analyzes the extracted text and classifies the data based on the type of document, such as into categories like "utility bill," "receipt," or "medical statement," using machine learning algorithms.

[1975] Step 6:

[1976] The server organizes the classified data as structured data and stores it in a database. The data is stored in fields such as date, amount, and item name.

[1977] Step 7:

[1978] A user enters a search query through an application, for example, "electricity bill details for the last 3 months."

[1979] Step 8:

[1980] The terminal sends the user's search query to the server.

[1981] Step 9:

[1982] The server searches the database for data corresponding to the search query, using indexes to efficiently extract the data.

[1983] Step 10:

[1984] The server creates a summary of the search results using a summary generation algorithm as needed.

[1985] Step 11:

[1986] The server transmits the generated search results and summaries to the terminal.

[1987] Step 12:

[1988] The terminal displays search results and summaries to the user, allowing the user to quickly access the information they need.

[1989] Step 13:

[1990] The server analyzes the user's past search history, frequency of use, access patterns, etc. Based on this, it applies an algorithm to recommend important information and actions.

[1991] Step 14:

[1992] The server recommends information that it deems important to the user, and this information is sent to the device as a notification.

[1993] Step 15:

[1994] The terminal displays a notification of the recommended information to the user and provides an interface that allows the user to view detailed information.

[1995] Step 16:

[1996] The server uses technology to recognize the user's emotions, for example, by analyzing the user's emotional state from their typing speed, context, facial recognition, etc.

[1997] Step 17:

[1998] The server recommends information and adjusts the summary of search results based on the user's recognized emotions. The emotion engine displays simple, essential information when the user is stressed, and provides detailed information when the user is relaxed.

[1999] Step 18:

[2000] The device displays the most appropriate information according to the user's emotional state, thereby realizing personalized information provision that matches the user's emotional state.

[2001] These are the processing steps of the system that combines the emotion engine. This system simplifies the management of paper documents for users, enables quick access to necessary information, and provides information based on emotions.

[2002] Example 2

[2003] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2004] In conventional paper document management systems, digitizing and searching documents is cumbersome, and the accuracy of summarization and information recommendations is low.In addition, information presentation does not take into account the user's emotional state, and the user experience is not optimized.

[2005] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2006] In this invention, the server includes means for capturing paper documents as digital images, character recognition means for converting the captured image data into character information, data classification means for classifying the character information and converting it into structured data, database means for saving the data converted by the data classification means, search input means for a user to input a search query, search result display means for searching the database based on the search query and displaying the results, and emotion recognition means for recognizing the user's emotion and adjusting the way information is presented. This allows users to efficiently digitize documents, easily search and acquire information, and receive information optimally suited to their emotional state.

[2007] "Means for capturing paper documents as digital images" refers to a device or software that allows a user to photograph a paper document using a smartphone or scanner and capture the image as a digital image.

[2008] "Character recognition means" refers to a technology for extracting character information from a digital image, specifically using OCR (optical character recognition) technology.

[2009] A "data classification means" is an algorithm or machine learning model that analyzes extracted text information and classifies it into specific categories.

[2010] "Database means" refers to a database system for structuring and storing classified data, including SQL and NoSQL databases.

[2011] A "search input means" is an application or system that provides an interface for a user to enter a search query in order to search for specific information.

[2012] The "search result display means" is an interface for displaying to the user the data acquired based on the search query, and displays it as text or graphs on the screen.

[2013] "Emotion recognition means" is a technology for analyzing the user's emotional state, and recognizes emotions using data such as input speed, context, and facial recognition.

[2014] MODE FOR CARRYING OUT THE INVENTION

[2015] The system of the present invention efficiently digitizes paper documents kept at home by users and manages, searches, summarizes, reports, and recommends information. It also includes an emotion engine that recognizes the user's emotional state and provides information accordingly. This system operates using the following hardware and software:

[2016] Hardware and Software Configuration

[2017] 1. Paper document capture method

[2018] Users use a smartphone or scanner to capture digital images of paper documents (such as electricity bills, shopping receipts, medical bills, etc.), and then upload these images to the device using a dedicated application.

[2019] 2. Character recognition means (OCR technology)

[2020] The server receives the digital image sent by the user and uses OCR (Optical Character Recognition) technology to extract text information from the image. Specifically, a library such as Tesseract OCR is used to perform preprocessing such as noise removal and tilt correction before recognizing the text.

[2021] 3. Data Classification Methods

[2022] The server analyzes the text extracted by OCR and determines the document type using machine learning algorithms such as SVM (support vector machine) and neural networks. For example, an electricity bill would be classified as a "utility bill" category.

[2023] 4. Database Means

[2024] The server stores the classified data in a database using a structured database such as MySQL or PostgreSQL, where the data is stored in the form of field name / value pairs, allowing for fast and efficient data searches and report generation.

[2025] 5. Search Input Methods

[2026] The user enters a search query into the application's search bar, for example, a specific query such as "electricity bill details for the last three months." This search query is sent to the server via the device.

[2027] 6. Search result display method

[2028] The server searches the database based on the received search query and extracts the relevant data. It then generates a summary from the extracted data and displays the results on the user's device. For example, a search result for "electricity bill details for the past three months" would display the details for each month and the total amount.

[2029] 7. Information recommendation means

[2030] The server recommends appropriate information based on the user's usage history and importance. For example, if an insurance contract is due for renewal, the server notifies the user of this information and recommends detailed information. This ensures that the user does not miss important information.

[2031] 8. Emotion recognition means

[2032] The server analyzes the user's emotional state based on their input speed, context, facial recognition, etc. For example, it can accurately recognize the user's emotional state by using Python's OpenCV library and various cloud APIs (e.g., Azure's Emotion API). If the input speed is slow or if words indicating stress are detected in the context, it is determined that the user is feeling stressed.

[2033] 9. Emotion Engine

[2034] The server adjusts the way information is presented based on the user's recognized emotions. For example, if the user is stressed, only simple and important information is displayed, while if the user is relaxed, detailed information and additional options are provided. This allows the server to provide optimal information according to the user's emotional state.

[2035] Examples of concrete examples and prompts

[2036] Example 1: Utility bill management

[2037] Users take a photo of their electricity bill with their smartphone and upload it to a dedicated app.

[2038] The device sends the uploaded image to the server.

[2039] The server uses OCR technology to extract text from the image and classify it into the "utility bills" category.

[2040] The server stores the data in a database so that it can be quickly accessed when a user searches.

[2041] Example 2: Searching for medical details

[2042] If a user wants to check the details of medical expenses for the past year, they enter the search query "medical expenses for the past year."

[2043] The terminal sends a search query to the server, which searches the database for the relevant medical details.

[2044] The server displays the search results and their summaries to the user.

[2045] Example 3: Creating a monthly report from receipts

[2046] If the user wants to know the total food expenses for the month, he / she inputs a request such as "Total food expenses for this month."

[2047] The terminal sends a request to the server, and the server aggregates the data for the specified period.

[2048] The server generates a report of the aggregated results and provides it to the user.

[2049] Example 4: Recommending insurance documents

[2050] The server analyzes the user's past search history and identifies important insurance-related information.

[2051] The server notifies the user that the time for renewal of the insurance contract is approaching and recommends related information to the user.

[2052] The user reviews the recommended information and takes the necessary action.

[2053] Example 5: Emotion-based information provision

[2054] The server recognizes the user's emotions, and if the user is feeling stressed, for example, it prioritizes displaying concise and important information.

[2055] If the user is relaxed, provide more information or additional options, which provides optimal information to the user.

[2056] Example prompts to input to the generative AI model

[2057] Please tell me your electricity bill details for the past three months.

[2058] "Calculate the total cost of food last month."

[2059] Please show me your medical expenses for the past year.

[2060] "Can you tell me when my insurance policy is due for renewal?"

[2061] "Show me recommendations based on my current mood."

[2062] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2063] Step 1:

[2064] The user takes a photo of the document and captures it as a digital image.

[2065] A user takes a photo of a paper document using a smartphone or scanner. For example, they take a photo of an electricity bill. Then, they use a dedicated application to upload the captured image file to their device. The input is a paper document, and the output is a digital image file.

[2066] Step 2:

[2067] The device sends the digital image to the server

[2068] The terminal transfers the digital image file uploaded by the user to the server. The input is the digital image file, and the output is the digital image file stored on the server. This completes the communication until the image reaches the server.

[2069] Step 3:

[2070] The server extracts text information using OCR technology.

[2071] The server analyzes the received digital image and extracts text information from the image using OCR technology. Specifically, it uses the Tesseract OCR library and performs preprocessing such as noise removal and tilt correction. The input is the digital image file, and the output is the extracted text information.

[2072] Step 4:

[2073] The server classifies the data

[2074] The server analyzes the text extracted by OCR and uses machine learning algorithms (such as SVM or neural networks) to determine the type of document and classify it. For example, an electricity bill is classified into the "utility bill" category. The input is text, and the output is classified data.

[2075] Step 5:

[2076] The server stores the classified data in a database.

[2077] The server stores the classified data in a database. Specifically, it uses a MySQL or PostgreSQL database and stores the data in a structured format of field name / value pairs. The input is the classified data, and the output is the data stored in the database.

[2078] Step 6:

[2079] The user enters a search query

[2080] A user enters a query into the application's search bar to search for specific information. For example, a specific search query such as "electricity bill details for the last three months." The input is the user's search query, and the output is the transmission of the search query from the device to the server.

[2081] Step 7:

[2082] The server searches the database and displays the results

[2083] The server searches the database based on the received search query and extracts the relevant data. A summary is generated from the extracted data and the results are displayed on the user's device. For example, a search result for "electricity bill details for the past three months" would display the details for each month and the total amount. The input is the search query, and the output is the display of the search results.

[2084] Step 8:

[2085] Server recommends information

[2086] The server recommends appropriate information based on the user's usage history and importance. For example, if an insurance contract is due for renewal, the server notifies the user of this information and recommends detailed information. This prevents the user from missing important information. The input is the user's usage history and importance, and the output is a notification of recommended information.

[2087] Step 9:

[2088] The server recognizes emotions and adjusts information

[2089] The server analyzes and recognizes the user's emotional state based on their input speed, context, and facial recognition. Specifically, it uses the Python OpenCV library and Azure's Emotion API. For example, if the input speed is slow or if the user uses many words indicating stress, it determines that the user is feeling stressed. The input is the user's input data and video data, and the output is the analysis result of the user's emotional state.

[2090] Step 10:

[2091] The server provides information using an emotion engine

[2092] The server adjusts the way information is presented based on the user's recognized emotions. For example, if the user is stressed, it displays only simple and important information, while if the user is relaxed, it provides detailed information and additional options. The input is the analysis result of the user's emotional state, and the output is the adjusted information provided.

[2093] (Application example 2)

[2094] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2095] In conventional paper document management systems, managing paper coupons and receipts is cumbersome, making it difficult to digitize them for easy user use. Furthermore, providing information according to the user's emotional state is not considered, which does not lead to an improved user experience. This has led to a demand for a system that allows users to efficiently manage paper information and quickly obtain the information they need without feeling stressed.

[2096] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2097] In this invention, the server includes means for capturing paper documents as digital images, character recognition means for converting the captured image data into character information, data classification means for classifying the character information and converting it into structured data, database means for saving the data converted by the data classification means, search input means for a user to input a search query, search result display means for searching the database based on the search query and displaying the results, emotion recognition means for recognizing the emotional state of the user, and emotion engine means for adjusting the method of providing information based on the recognized emotional state. This enables users to efficiently digitize and manage paper coupons and receipts, and further enables providing information according to the user's emotional state, thereby providing a more comfortable and personalized user experience.

[2098] - "Means for capturing paper documents" means means for capturing paper documents as digital images.

[2099] The "character recognition means" is a means for converting captured digital image data into character information.

[2100] The "data classification means" is a means for classifying the character information converted by the character recognition means into categories and converting it into structured data.

[2101] "Database Means" means a means for storing classified structured data and making it accessible as needed.

[2102] A "search input means" is a means by which a user inputs a search query.

[2103] The "search result display means" is a means for searching a database based on an input search query and displaying the results to the user.

[2104] The "emotion recognition means" is a means for analyzing and recognizing the user's emotional state.

[2105] An "emotion engine means" is a means for adjusting how information is presented based on the perceived emotional state of the user.

[2106] This invention describes a system for improving the shopping experience for brick-and-mortar stores that digitizes paper documents in the home and performs management, search, summarization, and information recommendation, in addition to a system that combines an emotion engine that recognizes user emotions.

[2107] System configuration and functions

[2108] This system has the following configuration and functions:

[2109] 1. Paper document capture method

[2110] This is a method for users to take a photo of a paper coupon or receipt using their smartphone camera and import it as a digital image. For example, a user can take a photo of a receipt they receive at a store cash register using a dedicated app.

[2111] 2. Character recognition method (OCR)

[2112] The server receives the digital image sent by the user and uses OCR technology to extract text information from the image. The Google Cloud Vision API is used to achieve highly accurate character recognition.

[2113] 3. Data Classification Methods

[2114] The server analyzes the extracted text and categorizes the data based on the type of coupon or receipt, using a machine learning algorithm to categorize the data into categories such as food, daily necessities, and services.

[2115] 4. Database Means

[2116] The server stores the classified data in the Firebase database and manages it as structured data, allowing users to quickly access the information they need.

[2117] 5. Search Input Methods

[2118] A way for a user to enter a search query into a search bar within an application, for example, "Show me this month's grocery receipts."

[2119] 6. Search result display method

[2120] It is a way for the server to search the Firebase database based on a search query and display the relevant data to the user, along with generating a summary of the search results.

[2121] 7. Information recommendation means

[2122] This is a means by which the server analyzes information based on the frequency and importance of the user's use and recommends appropriate coupons and products to the user. For example, it analyzes the user's purchase history and recommends coupons for products that the user is likely to purchase next.

[2123] 8. Emotion recognition means

[2124] The server uses technology to recognize the user's emotions. It uses Amazon Rekognition to analyze the user's emotional state from their typing speed and facial expressions.

[2125] 9. Emotional Engine Means

[2126] It is a way for the server to recommend information and tailor search result summaries based on the user's perceived emotions. The emotion engine can present simple, essential information when the user is stressed, and detailed information when the user is relaxed.

[2127] Specific examples

[2128] For example, if a user types "Show this month's grocery receipts" into a smartphone app, the following will happen:

[2129] 1. The user takes a photo of the receipt with their smartphone and uploads it to the app.

[2130] 2. The app sends the digital image to a server.

[2131] 3. The server uses the Google Cloud Vision API to extract text from the image.

[2132] 4. The server classifies the extracted text information into categories (food, daily necessities, etc.) and stores it in the Firebase database.

[2133] 5. The user uses a search input method to enter the search query "Show this month's grocery receipts."

[2134] 6. The server searches the database, provides the relevant receipt information in a search result display means, and generates a summary.

[2135] 7. Use Amazon Rekognition to analyze user sentiment and adjust how information is displayed using an emotion engine.

[2136] Prompt Sentence Examples

[2137] When the user enters "Show this month's food receipts," the server displays the corresponding receipt information, and can provide information according to the user's emotions.

[2138] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2139] Step 1:

[2140] The user takes a photo of a paper coupon or receipt with their smartphone.

[2141] Input: Paper coupons and receipts

[2142] Output: Coupons and receipts as digital images

[2143] How it works: The user takes a photo of a paper coupon or receipt using the camera function of the dedicated app. The captured image is saved to the smartphone's storage.

[2144] Step 2:

[2145] The terminal transmits the captured digital image to the server.

[2146] Input: The digital image acquired in step 1

[2147] Output: Digital image sent to server

[2148] How it works: The app sends the captured image to a cloud server, along with image metadata (such as the date and time of the photo and user information).

[2149] Step 3:

[2150] The server performs OCR processing to extract characters from the digital image.

[2151] Input: Digital image sent to the server

[2152] Output: Extracted text information

[2153] How it works: The server uses the Google Cloud Vision API to extract text from digital images, including preprocessing such as noise removal and deskewing.

[2154] Step 4:

[2155] The server classifies the extracted text information into categories.

[2156] Input: Character information extracted in step 3

[2157] Output: Classified text information

[2158] How it works: The server uses machine learning algorithms to classify the extracted text into categories such as coupons, receipts, food, and household items.

[2159] Step 5:

[2160] The server stores the classified character information in a database.

[2161] Input: Character information classified in step 4

[2162] Output: Text information stored in the database

[2163] How it works: The server stores classified textual information in the Firebase database. The stored data is managed as structured data, making it easy to search.

[2164] Step 6:

[2165] A user enters a search query into a search bar within the application.

[2166] Input: User's search query (e.g., "View this month's grocery receipts")

[2167] Output: The search query is sent to the server

[2168] What happens: A user types a query into an application's search bar, for example, "Show me this month's grocery receipts."

[2169] Step 7:

[2170] The server searches the database and retrieves the relevant data.

[2171] Input: The search query entered in step 6

[2172] Output: Data as search results

[2173] How it works: Your server searches your Firebase database to retrieve data that matches your query. For example, it aggregates relevant receipt information to display "Grocery Receipts of the Month."

[2174] Step 8:

[2175] The server generates and displays the search results and their summaries.

[2176] Input: Search results obtained in step 7

[2177] Output: Summarized search results

[2178] How it works: The server generates a summary based on the search results, for example aggregating multiple receipts and displaying the total amount and key purchase items.

[2179] Step 9:

[2180] The server recognizes the user's emotions and adjusts the display method according to that information.

[2181] Input: User's face image and input speed information

[2182] Output: How to display the adjusted information

[2183] How it works: The server uses Amazon Rekognition to analyze the user's facial image and typing speed to recognize their emotional state. For example, if the user is stressed, it will display simple, essential information, and if they are relaxed, it will provide detailed information.

[2184] By implementing the above steps, users can efficiently digitize and manage paper coupons and receipts, and quickly search and retrieve the information they need. Furthermore, the emotion engine makes it possible to provide personalized information based on the user's emotional state.

[2185] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2186] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2187] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2188] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2189] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2190] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2191] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2192] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2193] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2194] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2195] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2196] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2197] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2198] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2199] It is not necessary to store all of the spe...

Claims

1. a means for capturing a paper document as a digital image; character recognition means for converting captured image data into character information; a data classification means for classifying character information and converting it into structured data; database means for storing the data converted by the data classification means; a search input means for a user to input a search query; search result display means for searching a database based on the search query and displaying the results; A system including:

2. 2. The system according to claim 1, further comprising information recommendation means for recommending information based on frequency of use and importance.

3. 10. The system of claim 1, further comprising a summary generator for generating a summary of the search results.

Images