system

Abstract

Provide a system. An image capture means for capturing an image of a paper-based document; A means for converting the acquired image into text data using an optical character recognition engine; means for formatting the converted text data; A communication means for transmitting the formatted data to a server; A means for storing and linking the received data in a database; means for processing the stored data based on an analytical algorithm; a means of visualizing the results of the analysis; means for transmitting the visualized results to a user's terminal for display; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] It remains difficult for small and medium-sized enterprises and sole proprietors to easily digitize paper-based data and analyze and visualize it in real time. In particular, there is a lack of easy-to-use systems that efficiently convert paper-based documents into digital data, analyze it, and use it as information useful for business decision-making. Furthermore, existing solutions are complex to operate and expensive, making them difficult for small and medium-sized enterprises and sole proprietors to use. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides a system including an image capture means for capturing images of paper-based documents, a means for converting the captured images into text data using an optical character recognition engine, a means for formatting the converted text data, a communication means for transmitting the formatted data to a server, a means for storing and linking the received data in a database, a means for processing the stored data based on an analytical algorithm, a means for visualizing the analysis results, and a means for transmitting and displaying the visualized results to a user's terminal. This system enables users to easily digitize paper-based documents and analyze and visualize data in real time, thereby supporting fast and effective business decision-making.

[0006] "Image capture means" refers to a device or function for photographing and capturing a digital image of a paper-based document.

[0007] An "optical character recognition engine" is software or algorithms used to recognize characters in an image and convert them into digital text data.

[0008] "Text data" refers to textual and numerical information in digital form extracted by an optical character recognition engine.

[0009] A "format" is a standard or format for arranging text data into a specific structure or layout.

[0010] "Communications" refers to the technologies, protocols, and devices used to transmit data from one point to another.

[0011] A "server" is a computer system that stores, processes, and serves data over a network.

[0012] A "database" is a collection of information that is structured to efficiently manage stored data and make it easy to search and use.

[0013] An "analytical algorithm" is a mathematical method or set of computational steps for processing and analyzing data according to specific rules or procedures.

[0014] "Visualization" refers to displaying analytical results in the form of graphs, charts, diagrams, etc. so that they are easy for users to understand.

[0015] A "terminal" is an electronic device that a user uses to import, display, and operate data.

[0016] "User" means any person or entity that operates the System and uses the Services. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10]1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram illustrating a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention provides a system that allows small and medium-sized enterprises and sole proprietors to easily digitize paper-based data and analyze and visualize the data in real time. This system is implemented as follows.

[0039] 1. Data capture (camera capture of paper-based documents)

[0040] User:

[0041] Users use their smartphone camera to take a picture of a paper-based document (e.g., a sales ledger or expense receipt), after which the application automatically acquires the image and begins processing it.

[0042] Device:

[0043] The captured image is converted into digital text by the device's optical character recognition (OCR) engine, which then formats the text data according to the user's specifications. This formatting process includes categorizing the data by date, amount, category, and other items.

[0044] 2. Data transmission and storage

[0045] Device:

[0046] The formatted data is encrypted and sent securely from the device to the server using the HTTPS protocol.

[0047] server:

[0048] The data received by the server is first decrypted and then stored in a database based on the user's account information. During the storage process, the data is checked for integrity to ensure that the information registered in the database is accurate.

[0049] 3. Data analysis and visualization

[0050] server:

[0051] The stored data is retrieved periodically or upon user request and predefined analytical algorithms are applied to it, for example to analyze sales trends or popular products. This is done by using SQL queries to retrieve data from the database and then processing it with analytical tools such as Python or R.

[0052] server:

[0053] The results of the analysis are generated in a visually understandable format, such as graphs or charts, using graphing libraries such as Matplotlib or D3.js.

[0054] 4. Sending and displaying results

[0055] server:

[0056] The visualized results data is sent to the user's mobile device in JSON format and is notified to the user via push notification or email.

[0057] Device:

[0058] The visualized data received by the device is displayed in a dedicated application, and users can view this data through the application's interface and quickly check the information they need.

[0059] 5. Utilizing the analysis results

[0060] User:

[0061] Users can make business decisions based on visualized data, such as adjusting inventory based on sales trends or analyzing the distribution of expenses to consider cost-cutting measures.

[0062] Specific examples

[0063] For retailers

[0064] The user, a retail store owner, takes a photo of their daily sales ledger with their smartphone camera. The device sends the image to an OCR engine, which converts and formats it into sales data. This data is securely sent to a server, which stores and analyzes it in a database. The analysis results are visualized as a sales trend graph and displayed on the user's smartphone. The owner can then use this data to adjust their inventory management and purchasing plans.

[0065] For sole proprietors

[0066] Self-employed individuals take photos of expense receipts with their smartphone camera, and the device converts the images into text data using an OCR engine. The converted and formatted expense data is sent to a server and stored in a database. The server analyzes the distribution of expenses by category and sends the results, visualized as a pie chart, to the user's device. Self-employed individuals can use this data to review their expenses and devise cost-cutting measures.

[0067] This allows users to quickly extract necessary information from paper-based data and use the present invention as a useful tool for making business decisions.

[0068] The processing flow will be explained below.

[0069] Step 1:

[0070] The user launches the camera app on their smartphone and takes a photo of a paper-based document (e.g., a sales ledger or expense receipt). The user checks the appropriate position and lighting, then presses the shutter button.

[0071] Step 2:

[0072] The device sends the captured image to an optical character recognition (OCR) engine, which extracts the characters in the image as digital text by analyzing their shapes.

[0073] Step 3:

[0074] The device then formats the text data converted by OCR into a specific format, which involves organizing the data based on items such as date, amount, and category, and converting it into the appropriate format.

[0075] Step 4:

[0076] The device encrypts the formatted data and sends it to the server. This communication uses the HTTPS protocol, ensuring secure transmission of data.

[0077] Step 5:

[0078] The server receives the data, decrypts it, associates it with a specific user account, and stores it in a database, where the data is verified for integrity and placement in the correct location.

[0079] Step 6:

[0080] The server retrieves the data stored in the database periodically or upon user request and applies predefined analytical algorithms, which may include, for example, calculating sales trends or expense distributions.

[0081] Step 7:

[0082] The server generates the analysis results in a visually understandable format using graphing libraries such as Matplotlib or D3.js, for example, to create bar charts, pie charts, and line graphs.

[0083] Step 8:

[0084] The server sends the generated visualization data to the user's device in JSON format, and the user is notified via push notification or email.

[0085] Step 9:

[0086] The device analyzes the received visualization data and displays it in a dedicated application, which the user can use to view graphs and charts through the application interface.

[0087] Step 10:

[0088] Users can make business decisions based on the displayed data, such as adjusting inventory based on sales trends or considering cost-cutting measures based on expense distribution.

[0089] Example 1

[0090] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0091] Currently, digitizing paper-based documents and analyzing and visualizing data quickly and accurately is a burden for small and medium-sized enterprises and sole proprietors. In particular, manual data entry and compilation requires time and effort, which delays business decision-making. Furthermore, appropriate measures to ensure data security and integrity are required.

[0092] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0093] In this invention, the server includes an image acquisition means, a means for converting acquired text data into a specified format using an optical character recognition engine, a means for decrypting encrypted data and storing it in a database, a means for applying an analysis algorithm based on the stored data and processing it, a means for visualizing the analysis results as graphs and charts, and a means for transmitting the visualized result data to a user's terminal and displaying it. This enables small and medium-sized enterprises and sole proprietors to quickly and accurately digitize paper-based data and analyze and visualize the data in real time.

[0094] "Paper-based document" refers to any document containing data or information recorded on physical paper.

[0095] "Image capture means" refers to a device or technology for capturing a digital image of a paper-based document.

[0096] An "optical character recognition engine" refers to software or technology that recognizes characters from images and converts them into digital text.

[0097] "Text data" refers to data in which character information is stored electronically.

[0098] "Means for formatting" refers to devices or technologies for arranging acquired text data to conform to a specified format or structure.

[0099] "Communications" refers to the infrastructure and technology used to transmit data to other devices or systems.

[0100] "Encryption" refers to the technology of converting data using a certain algorithm to protect it.

[0101] "Server" means a computing device that stores and processes data and responds to requests from clients.

[0102] A "database" refers to a system for efficiently storing, retrieving, and managing structured data.

[0103] "Analytical algorithm" refers to a set of methods or procedures used to analyze data and extract specific information or trends.

[0104] "Visualization" refers to the technique of representing data in a visual format such as a graph or chart.

[0105] "User's device" refers to a device such as a computer or smartphone used by a user.

[0106] "Means of transmission" refers to the technology or method used to deliver data to a specified destination.

[0107] "Analysis Results" refers to the information and trends obtained by the analysis algorithm.

[0108] The present invention provides a system that allows small and medium-sized enterprises and sole proprietors to easily digitize paper-based data and analyze and visualize the data in real time. This system is implemented as follows.

[0109] Data capture (camera capture of paper-based documents)

[0110] User:

[0111] Users use their smartphone camera to take a photo of a paper document such as a sales ledger or expense receipt, after which a dedicated application automatically acquires the image and moves on to the next step.

[0112] Device:

[0113] The device passes the captured image to the Tesseract OCR engine, which extracts text data from the image. The extracted text data is then formatted by date, amount, category, and other items. For example, it can be formatted as "2023-10-01, Sales, 10,000 yen."

[0114] Data transmission and storage

[0115] Device:

[0116] The formatted data is encrypted using the AES encryption algorithm and sent to the server via the HTTPS protocol.

[0117] server:

[0118] The server decrypts the received data and stores it in the PostgreSQL database based on the user's account information. The data storage process checks for integrity and ensures accurate information is registered in the database.

[0119] Data Analysis and Visualization

[0120] server:

[0121] The server retrieves the stored data periodically or upon user request and applies analytical algorithms such as Pandas or R to it, for example, to analyze sales trends or popular products.

[0122] server:

[0123] The results of the analysis are generated as graphs and charts using visualization libraries such as Matplotlib and D3.js, for example a line graph showing how sales data fluctuates from month to month.

[0124] Sending and displaying results

[0125] server:

[0126] The visualized results data is sent in JSON format to the user's mobile device and notified to the user via push notification or email.

[0127] Device:

[0128] The device displays the received visualized data using a dedicated application, allowing users to check graphs and charts and quickly obtain the information they need.

[0129] Utilizing analysis results

[0130] User:

[0131] Users can make business decisions based on visualized data, such as adjusting inventory based on sales trends or analyzing the distribution of expenses to consider cost-cutting measures.

[0132] Specific examples

[0133] For retailers

[0134] A retail store owner takes a photo of their daily sales ledger using their smartphone camera. The device sends the image to the Tesseract OCR engine, which converts and formats it into sales data. This data is AES encrypted and sent to the server via HTTPS. The server decrypts the data and stores it in a PostgreSQL database. The Pandas library then calculates monthly sales trends and creates a line graph using Matplotlib. The created graph is sent in JSON format to the user's smartphone. The user receives a push notification, checks the graph, and adjusts inventory management and purchasing plans.

[0135] For sole proprietors

[0136] Self-employed individuals take a photo of their expense receipts with their smartphone camera. The device analyzes the image using the Tesseract OCR engine, extracts expense data, and formats it by category. The formatted expense data is AES encrypted and sent to the server via HTTPS. The server decrypts the data and stores it in a PostgreSQL database. The server analyzes the distribution of expenses by category using the R programming language and generates a pie chart using D3.js. The generated pie chart is sent in JSON format to the user's smartphone, and a push notification is sent. The user can then review the pie chart and devise measures to reduce expenses.

[0137] Example prompts for generative AI models

[0138] Prompt statement:

[0139] "I want to take a photo of my sales ledger with my smartphone, process it using OCR, and convert it into digital data for analysis. How can I create a graph that visually shows sales trends?"

[0140] This allows users to quickly extract necessary information from paper-based data and use the present invention as a useful tool for making business decisions.

[0141] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0142] Step 1:

[0143] User:

[0144] Users use their smartphone's camera to take pictures of paper-based documents such as sales ledgers and expense receipts.

[0145] Input: Paper-based documents (physical ledger books and receipts).

[0146] Output: Early digital images.

[0147] What it does: A user opens their smartphone camera, frames a document, and presses the shutter button, and the image is then sent to a dedicated application.

[0148] Step 2:

[0149] Device:

[0150] The device passes the captured image to the Tesseract OCR engine, which extracts text data from the image.

[0151] Input: Early digital images.

[0152] Output: Raw text data.

[0153] How it works: The dedicated application sends the image to the OCR engine, which analyzes the characters in the image and generates text data. Text data in the format "2023-10-01, Sales, 10,000 yen" is output.

[0154] Step 3:

[0155] Device:

[0156] Formats the text data obtained from the OCR engine into the specified format.

[0157] Input: Raw text data.

[0158] Output: Formatted text data.

[0159] What it does: The algorithm parses the text data and separates it into fields such as date, amount, category, etc. The formatted data is then formatted as "YYYY-MM-DD, category, amount".

[0160] Step 4:

[0161] Device:

[0162] The formatted data is encrypted using the AES encryption algorithm.

[0163] Input: Formatted text data.

[0164] Output: The encrypted data.

[0165] What happens: The application invokes the AES encryption algorithm to encrypt the formatted data, which is then ready to be sent to the server via the HTTPS protocol.

[0166] Step 5:

[0167] Device:

[0168] The encrypted data is sent to the server via the HTTPS protocol.

[0169] Input: Encrypted data.

[0170] Output: The encrypted data sent.

[0171] What happens: Your device sends encrypted data to the server using the HTTPS protocol, where it arrives securely.

[0172] Step 6:

[0173] server:

[0174] The server decrypts the received data using AES encryption.

[0175] Input: The encrypted data sent.

[0176] Output: The decrypted data.

[0177] What happens: The server uses the AES decryption algorithm to decrypt the data and return it to the original text, ready for database entry.

[0178] Step 7:

[0179] server:

[0180] The decrypted data is stored in a database based on the user's account information.

[0181] Input: Decrypted data.

[0182] Output: Data stored in a database.

[0183] What happens: The server establishes a database connection, associates the decrypted data with the corresponding user account, and stores it. The data is checked for integrity when it is stored.

[0184] Step 8:

[0185] server:

[0186] The stored data is retrieved periodically or upon user request and analytical algorithms are applied.

[0187] Input: Data stored in a database.

[0188] Output: Analysis results.

[0189] What it does: The server retrieves data using SQL queries and processes it using tools like Pandas and R. For example, it applies an algorithm to analyze sales trends and calculates the resulting increase or decrease in sales.

[0190] Step 9:

[0191] server:

[0192] The analysis results are generated as graphs and charts using visualization libraries such as Matplotlib and D3.js.

[0193] Input: Analysis results.

[0194] Output: Visualized data (graphs and charts).

[0195] What happens: The server calls a visualization library to create, for example, a line graph showing monthly sales trends. The visualization is generated and ready to send to the user.

[0196] Step 10:

[0197] server:

[0198] The visualized result data is sent in JSON format to the user's mobile device and the user is notified via push notification or email.

[0199] Input: The visualized data.

[0200] Output: The submitted visualization data.

[0201] Specific operation: The server converts the data into JSON format and sends the result to the user via push notification or email. The user's device receives the data.

[0202] Step 11:

[0203] Device:

[0204] The terminal displays the received visualization data using a dedicated application, allowing the user to check graphs and charts.

[0205] Input: The submitted visualization data.

[0206] Output: The displayed graph or chart.

[0207] How it works: A dedicated application analyzes the data and provides visualized information to the user, allowing the user to quickly check the information they need through the application.

[0208] These steps enable users to quickly and accurately digitize paper-based documents and analyze and visualize data in real time.

[0209] (Application example 1)

[0210] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0211] Currently, small and medium-sized enterprises and sole proprietors face many challenges in quickly converting paper-based sales slips and expense receipts into digital data and using that data to make business decisions. Specifically, manual data entry requires time and effort and is prone to human error. Furthermore, real-time data analysis is difficult, making it difficult to immediately identify sales trends and wasteful expenses. This delays the optimization of inventory management and purchasing plans, as well as the development of cost-cutting measures. Therefore, there is a need for a system that can easily digitize this paper-based data and analyze and visualize it in real time.

[0212] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0213] In this invention, the server includes means for sending visualized results to the user's terminal and displaying them, means for processing the stored data based on an analytical algorithm, and means for storing and linking the received data in a database. This makes it possible to convert photos of sales slips and expense receipts taken by the user into digital data and visualize the analysis results, including sales trends and popular products, in real time.

[0214] "Paper-based documents" are records of information written on paper, such as sales slips and expense receipts.

[0215] The "image acquisition means" is a means for taking a photograph of a paper-based document using a camera such as a smartphone and acquiring the image.

[0216] An "optical character recognition engine" is software that extracts text from captured images and converts it into digital data.

[0217] The "means for formatting" is a means for classifying the converted text data into date, amount, category, etc., and formatting it into a predetermined format.

[0218] "Communication means" refers to a means for transmitting formatted data to a server via the Internet.

[0219] A "database" is a collection of digital data stored on a server, where each piece of data is appropriately tagged and linked.

[0220] An "analysis algorithm" is a calculation procedure used to analyze sales trends and popular products based on stored data.

[0221] "Means for visualizing analytical results" refers to means for visually representing the results obtained by analytical algorithms as graphs or charts.

[0222] The "user terminal" is a device such as a smartphone or tablet, which is used to display visualized analysis results.

[0223] "Sales trend" is an indicator that shows fluctuations in sales data over a certain period of time.

[0224] "Popular products" is an indicator of products with high sales during a specific period.

[0225] "Real-time" refers to the user instantly capturing and analyzing ongoing data.

[0226] This invention is a system that enables small and medium-sized enterprises and sole proprietors to easily digitize paper-based sales slips and expense receipts, and then analyze and visualize the data in real time. This system is composed of an image acquisition means, an optical character recognition engine, a data formatting means, a communication means, an analysis algorithm, and a means for visualizing the analysis results.

[0227] Hardware and software used

[0228] 1. Smartphone (device)

[0229] Image acquisition method:

[0230] Users use their smartphone to take a photo of a paper sales slip or expense receipt, and the image is captured directly within the application.

[0231] Optical Character Recognition Engine (OCR):

[0232] On the smartphone, the acquired image is converted into text data using an OCR engine (e.g., Tesseract).

[0233] 2. Server

[0234] Data formatting methods:

[0235] The converted text data is formatted by items such as date, amount, category, etc. before being sent to the server. The formatting process is carried out using a Python script on the smartphone.

[0236] Communication Method:

[0237] The formatted data is sent securely to the server using the HTTPS protocol.

[0238] Storage and linking methods:

[0239] The data received by the server is stored in a database and linked to the user's identification information. This database may be, for example, MySQL (registered trademark).

[0240] Analysis algorithm:

[0241] Algorithms are applied to the data stored in the database to analyze sales trends and popular products, using Python or R.

[0242] Analysis results visualization methods:

[0243] The analyzed results are visualized using graphing libraries such as Matplotlib and D3.js.

[0244] 3. User's Device

[0245] Viewing Results:

[0246] The resulting data is analyzed and visualized on the server and sent to the user's smartphone in JSON format, where it is displayed to the user through a dedicated application.

[0247] Specific examples

[0248] 1. For retailers:

[0249] Retail store owners take photos of their daily sales slips with their smartphone camera. The images are converted into text data using an OCR engine, and then formatted into fields such as date, amount, and category. This data is securely sent to a server for storage and analysis. The analysis results are visualized as sales trends and can be viewed in real time on the user's smartphone.

[0250] 2. For sole proprietors:

[0251] Self-employed individuals take photos of expense receipts with their smartphone camera, and the device converts the images into text data using an OCR engine. This data is then formatted and sent to a server. The analysis results on the server are visualized as expense distribution by category and displayed on the user's smartphone. Self-employed individuals can use this data to plan cost-cutting measures.

[0252] Prompt Sentence Examples

[0253] You can enter a generative AI model using a prompt like this:

[0254] Please generate a concrete example application based on the following invention. The invention is a system that allows small and medium-sized enterprises and sole proprietors to easily digitize paper-based data and analyze and visualize the data in real time. This application is installed on a smartphone and takes photos of sales slips, expense receipts, etc., digitizes them, and then analyzes and visualizes the data based on that.

[0255] Although the embodiments for carrying out the present invention have been described in detail above, the scope of the present invention is not limited to the above examples.

[0256] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0257] Step 1:

[0258] A user takes a photo of a paper-based sales slip or expense receipt with their smartphone camera. This action generates image data. The input is a paper-based document, and the output is image data. Specifically, the user launches the smartphone's camera app and takes a photo of the document.

[0259] Step 2:

[0260] Once the captured image data is saved on the device, the device uses its internal optical character recognition engine (OCR) to convert this image data into text data. The input is image data, and the output is text data. Specifically, the OCR engine (e.g., Tesseract) recognizes the characters in the image and extracts them as text.

[0261] Step 3:

[0262] The converted text data is formatted by a program on the terminal based on the date, amount, category, etc. The input is the converted text data, and the output is the formatted data. Specifically, the formatting algorithm analyzes the text data and separates it into fields.

[0263] Step 4:

[0264] The formatted data is sent to the server in encrypted form using the HTTPS protocol. The input is the formatted data, and the output is the encrypted data transmission completion. Specifically, the terminal opens a secure communication channel and sends the data to the endpoint.

[0265] Step 5:

[0266] The data received by the server is first decrypted, linked to the user's account information, and then stored in a database. The input is encrypted data, and the output is digital data stored in the database. Specifically, the server decodes the data, links it to the corresponding user information, and stores it in the database.

[0267] Step 6:

[0268] The stored data is processed based on analytical algorithms on the server. For example, sales trends and popular products are analyzed. The input is digital data stored in a database, and the output is the analysis results. Specifically, Python or R scripts use SQL queries to retrieve the data and perform the analysis.

[0269] Step 7:

[0270] The analysis results are generated as graphs and charts using a visualization library (e.g., Matplotlib or D3.js). The input is the analysis results, and the output is the visualized data. Specifically, various diagrams and charts are generated based on the analysis results.

[0271] Step 8:

[0272] The visualized result data is sent from the server to the user's device in JSON format. The input is the visualized data, and the output is the completion of sending the JSON format data. Specifically, the server sends the data to the user's device via push notification or email.

[0273] Step 9:

[0274] The data received by the user's device is displayed by a dedicated application. The input is JSON formatted data, and the output is visualized data displayed on the user interface. Specifically, the application parses the JSON data and displays it in an appropriate format.

[0275] Step 10:

[0276] Based on this visualized data, users make business decisions such as inventory management, adjusting purchasing plans, and planning cost reduction measures. The input is the visualized data, and the output is the user's decision. Specifically, users check the data on the application and take action as needed.

[0277] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0278] The present invention combines a system that allows small and medium-sized enterprises and sole proprietors to easily digitize paper-based data, analyze and visualize the data in real time, and further combines it with an emotion engine that recognizes the user's emotions. This system is implemented as follows.

[0279] 1. Data capture (camera capture of paper-based documents)

[0280] User:

[0281] The user launches the camera app on their smartphone, takes a photo of a paper-based document (e.g., a sales ledger or expense receipt), checks the appropriate position and lighting, and presses the shutter button.

[0282] Device:

[0283] The captured image is converted into digital text by the device's optical character recognition (OCR) engine, which then formats the text into a specified format. This formatting process includes sorting the data by date, amount, category, and other items.

[0284] 2. Data transmission and storage

[0285] Device:

[0286] The formatted data is encrypted and sent securely from the device to the server using the HTTPS protocol.

[0287] server:

[0288] Once the data arrives at the server, it is first decrypted and then stored in a database based on the user's account information. During the storage process, the data is verified for integrity and placement in the correct location.

[0289] 3. Data analysis and visualization

[0290] server:

[0291] Data stored in a database is retrieved periodically or upon user request and predefined analytical algorithms are applied, for example, to analyze sales trends or expense distribution.

[0292] server:

[0293] The analysis results are generated in a visually friendly format, using graphing libraries such as Matplotlib and D3.js to create bar charts, pie charts, line graphs, etc.

[0294] 4. Sending and displaying results

[0295] server:

[0296] The generated visualization data is sent to the user's device in JSON format, and the user is notified via push notification or email.

[0297] Device:

[0298] The received visualization data is displayed in a dedicated application, and users can view this data through the application interface and quickly check the information they need.

[0299] 5. Use of Emotion Engine

[0300] Device:

[0301] The emotion engine recognizes the user's facial expressions and voice, and acquires the user's emotion data in real time. The emotion engine uses machine learning algorithms to identify emotions (e.g., joy, anger, sadness, surprise).

[0302] server:

[0303] The acquired emotional data is incorporated into the analysis results, and the information provided takes into account the user's emotional state. For example, if the user expresses dissatisfaction, the system will provide a detailed explanation or additional advice.

[0304] Specific examples

[0305] For retailers

[0306] The user, a retail store owner, takes a photo of their daily sales ledger with their smartphone camera. The device sends the image to an OCR engine, which converts and formats it into sales data. This data is securely sent to a server, where it is stored and analyzed in a database. The analysis results are visualized as a sales trend graph and displayed on the user's smartphone. The emotion engine analyzes the owner's facial expressions, and if a positive emotion is recognized, the system shares success stories and provides further advice.

[0307] For sole proprietors

[0308] Self-employed individuals take photos of expense receipts with their smartphone camera, and the device converts and formats the images into text data using an OCR engine. The converted and formatted expense data is sent to a server and stored in a database. The server analyzes the distribution of expenses by category and sends the results, visualized as a pie chart, to the user's device. If the emotion engine recognizes the user's stress or anxiety, content offering specific ways to reduce costs and mitigation measures is displayed.

[0309] This allows users to receive not just data analysis but also emotionally-based advice, helping them make more effective business decisions.

[0310] The processing flow will be explained below.

[0311] Step 1:

[0312] The user launches the camera app on their smartphone and takes a photo of a paper-based document (e.g., a sales ledger or expense receipt). The user checks the appropriate position and lighting, then presses the shutter button.

[0313] Step 2:

[0314] The device sends the captured image to an optical character recognition (OCR) engine, which extracts the characters in the image as digital text. The OCR engine analyzes the characters in the image and converts the data into text.

[0315] Step 3:

[0316] The device then formats the text data converted by OCR into a specific format, which involves sorting the data by date, amount, category, etc., and converting it into the appropriate format.

[0317] Step 4:

[0318] The device encrypts the formatted data and sends it securely to the server, using the HTTPS protocol to ensure secure data transmission.

[0319] Step 5:

[0320] The server receives the data, decrypts it, associates it with a specific user account, and stores it in a database, where the data is verified for integrity and placement in the correct location.

[0321] Step 6:

[0322] The device transmits the user's facial expressions and voice to the emotion engine, which uses machine learning algorithms to identify the user's emotions (e.g., joy, anger, sadness, surprise) in real time.

[0323] Step 7:

[0324] The server retrieves business data stored in the database periodically or upon user request and applies predefined analytical algorithms to it, for example, analyzing sales trends or expense distribution.

[0325] Step 8:

[0326] The server integrates the analysis results with the user's emotional data and generates the analysis results in a visually easy-to-understand format, such as a bar graph, pie chart, or line graph.

[0327] Step 9:

[0328] The server sends the generated visualization data to the user's device in JSON format, and the user is notified via push notification or email.

[0329] Step 10:

[0330] The device analyzes the received visualization data and displays it in a dedicated application. Users can view graphs and charts through the application interface and quickly obtain the information they need.

[0331] Step 11:

[0332] The server dynamically adjusts the display content based on the user's emotional data. For example, if the user is feeling anxious, the analysis results will include detailed explanations and advice.

[0333] Step 12:

[0334] Users can make business decisions based on the displayed data and additional advice. For example, they can manage inventory based on sales trends or consider cost-cutting measures based on expense distribution. The system also includes advice based on emotional data, allowing users to make effective decisions while reducing psychological burden.

[0335] Example 2

[0336] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0337] For small and medium-sized enterprises and sole proprietors, digitizing paper-based data and analyzing and visualizing it in real time is a time-consuming and labor-intensive task. Furthermore, conventional systems lack the functionality to provide personalized advice that takes into account the user's emotions. As a result, users are limited to simply visualizing the data, making it difficult for them to make effective, emotion-based decisions.

[0338] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an image acquisition means for capturing a paper-based document, a means for converting the captured image into text data using an optical character recognition engine, a means for formatting the converted text data into a specific format, a communication means for transmitting the formatted data to a remote server, a means for storing the received data in a database and associating it with user information, a means for processing the stored data based on an analysis algorithm, a means for visually displaying the analysis results, a means for transmitting the visualized results to the user's device and displaying them, a means for using an emotion recognition engine for analyzing the user's emotions, and a means for including emotion data in the analysis results and providing information tailored to the user. This allows the user to easily digitize paper-based data, analyze and visualize the data in real time, and receive personalized advice based on their emotions.

[0339] "Paper-based documents" refers to text or data that is written on physical paper.

[0340] "Image capture means" refers to a device or method for capturing a paper-based document as a digital image.

[0341] "Optical character recognition engine" refers to software or algorithms that analyze character information from image data and convert it into digital text.

[0342] "Formatting means" refers to processing means that rearranges acquired digital text into a particular format or structure to facilitate storage in a database.

[0343] "Communication means" refers to the network protocols and devices used to transmit data to a remote server.

[0344] "Server" refers to a computer system for storing, processing, and managing data.

[0345] "Database" refers to a data storage system for efficiently storing, retrieving, and managing structured data.

[0346] An "analytical algorithm" refers to a set of computational procedures or rules for processing stored data and extracting useful information.

[0347] "Visualization means" refers to tools and methods for displaying extracted information in an easy-to-understand format, such as graphs or charts.

[0348] "User device" refers to a device (e.g., a smartphone or tablet) that a user owns and uses to display and interact with information.

[0349] An "emotion recognition engine" refers to software or algorithms that analyze and identify emotions from a user's facial expressions and voice.

[0350] "Emotion data" refers to data that indicates the emotional state of the user analyzed by an emotion recognition engine.

[0351] "Information provision" refers to the act of presenting necessary information or advice to a user.

[0352] This invention combines a system that allows small and medium-sized enterprises and individual business owners to easily digitize paper-based data, analyze and visualize the data in real time, and an emotion engine that recognizes user emotions. This system is realized by a specific combination of hardware and software.

[0353] Data capture (camera capture of paper-based documents)

[0354] User:

[0355] A user starts a camera app on a smartphone and takes a picture of a paper-based document (e.g., a sales ledger or expense receipt). For example, the user takes a picture of the sales ledger to obtain a digital image.

[0356] Device:

[0357] The device receives the captured image data and converts it into digital text using an optical character recognition engine (e.g., Google® Cloud Vision or Tesseract OCR), then formats the text data into a specific format (date, amount, category, etc.) using Python's Pandas library.

[0358] Data transmission and storage

[0359] Device:

[0360] The formatted data is encrypted and sent to the server using the HTTPS protocol. The SSL / TLS protocol is used for encryption. For example, dates, sales data, etc. are sent securely.

[0361] server:

[0362] The received data is decrypted and stored in a database (e.g., PostgreSQL) based on the user's account information using a web framework such as Django or Flask. The data is verified for consistency and placement in the appropriate fields.

[0363] Data Analysis and Visualization

[0364] server:

[0365] The stored data is retrieved periodically or upon user request and predefined analytical algorithms are applied using machine learning libraries such as Scikit-learn and TENSORFLOW®, for example to analyze sales trends or expense distribution.

[0366] server:

[0367] The analysis results are generated in visually easy-to-understand formats (bar graphs, pie charts, line graphs) using graphing libraries such as Matplotlib and D3.js.

[0368] Sending and displaying results

[0369] server:

[0370] The generated visualization data is sent to the user's device in JSON format, and a message such as "Monthly sales report has been generated" is provided via push notification or email.

[0371] Device:

[0372] The received data is displayed in a dedicated application using frameworks such as React Native or Flutter (registered trademark), and users can view graphs and charts through the app interface.

[0373] Using the Emotion Engine

[0374] Device:

[0375] The system captures the user's facial expressions and voice in real time using Microsoft® Azure®'s Face API and Google Cloud Speech-to-Text API, allowing it to analyze the user's emotions (e.g., joy, anger, sadness, surprise).

[0376] server:

[0377] The acquired emotional data is then incorporated into the analysis results to provide personalized advice and information based on the user's emotional state. For example, the system may provide a message such as, "You seem pleased with your recent sales growth. Would you like to share your success story?"

[0378] Specific examples

[0379] For retailers

[0380] User:

[0381] A retail store owner takes a photo of their daily sales ledger using their smartphone camera.

[0382] Device:

[0383] The device sends the image to the OCR engine, which converts and formats it into sales data.

[0384] server:

[0385] The data stored in the database is analyzed and visualized as a sales trend graph. The visualization results are sent to the user's smartphone and displayed in a dedicated application.

[0386] Emotion Engine:

[0387] The emotion engine analyzes the user's facial expressions, and if a positive emotion is recognized, it will share success stories and provide further advice.

[0388] For sole proprietors

[0389] User:

[0390] A self-employed person takes a photo of an expense receipt with their smartphone camera.

[0391] Device:

[0392] The device converts and formats the image into text data using an OCR engine.

[0393] server:

[0394] The data stored in the database is analyzed, and the distribution of expenses by category is visualized as a pie chart and sent to the user's device.

[0395] Emotion Engine:

[0396] The emotion engine recognizes the user's stress and anxiety and displays content offering specific ways to reduce costs and provide relief.

[0397] Prompt Sentence Examples

[0398] By inputting the following prompt sentence into the generative AI model, an explanation of the system design and processing described above can be generated.

[0399] "Design a system that allows small and medium-sized businesses and sole proprietors to easily digitize paper-based data and analyze and visualize the data in real time. This system should also incorporate an emotion engine that recognizes the user's emotions. Explain the system's processing in natural language by following the steps and providing concrete examples."

[0400] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0401] Step 1: Data capture (camera capture of paper-based documents)

[0402] User:

[0403] A user launches the camera app on their smartphone and takes a picture of a paper-based document (e.g., a sales ledger or expense receipt), checks whether the image is clear, and presses the shutter button. The input is the paper-based document, and the output is a digital image.

[0404] Device:

[0405] The device saves the captured image data and launches an OCR engine (such as Google Cloud Vision or Tesseract OCR). The input is a digital image, and the output is text data, which includes the date and amount.

[0406] Step 2: Formatting the text data

[0407] Device:

[0408] The text data obtained by the OCR engine is formatted using Python's Pandas library. The input is raw text data, and the output is formatted data, for example, "Date: 2023 / 10 / 01, Sales: 20,000 yen." Here, the text data is classified into specific items (date, amount, category, etc.).

[0409] Step 3: Sending data

[0410] Device:

[0411] The formatted data is encrypted using AES and sent to the server using the HTTPS protocol. The input is the formatted data and the output is the encrypted data. The SSL / TLS protocol is used for transmission.

[0412] Step 4: Decrypt and save the data

[0413] server:

[0414] Receives encrypted data and first decrypts it. The input is the encrypted data and the output is the decrypted data. Next, using a web framework such as Django or Flask, the data is saved in a database (e.g., PostgreSQL) based on the user's account information. The input is the decrypted data and the output is the data saved in the database.

[0415] Step 5: Acquire and analyze data

[0416] server:

[0417] The stored data is retrieved periodically or upon user request. The input is the data stored in the database, and the output is the retrieved data. Then, using a machine learning library such as Scikit-learn or TensorFlow, a predefined analytical algorithm is applied. For example, to analyze sales trends or expense distribution. The input is the retrieved data, and the output is the analysis results.

[0418] Step 6: Visualize the data

[0419] server:

[0420] The analysis results are generated in a visually easy-to-understand format (bar graph, pie chart, line graph) using graph creation libraries such as Matplotlib and D3.js. The input is the analysis results, and the output is the visualized data.

[0421] Step 7: Send and display visualization data

[0422] server:

[0423] The generated visualization data is sent to the user's device in JSON format. The input is the visualization data, and the output is the transmitted data. A message such as "Monthly sales report has been generated" is provided via push notification or email.

[0424] Device:

[0425] The received data is displayed in a dedicated application using a framework such as React Native or Flutter. The input is the transmitted JSON data, and the output is a graph or chart.

[0426] Step 8: Emotion recognition and advice provision

[0427] Device:

[0428] The user's facial expressions and voice are acquired in real time using Microsoft Azure's Face API and Google Cloud Speech-to-Text API. The input is the user's facial expressions and voice, and the output is emotion data. The emotion data is sent to the server upon request.

[0429] server:

[0430] Receives emotion data and incorporates it into the analysis results. For example, if the user is satisfied, relevant advice is provided. The input is emotion data, and the output is the advice the user applies. The advice is sent to the user's device in JSON format and displayed in a React Native or Flutter application.

[0431] (Application example 2)

[0432] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0433] Currently, many small and medium-sized enterprises and sole proprietors handle a large amount of paper-based documents, and digitizing this data is a time-consuming and labor-intensive process. While there is also a need for analysis and visualization to effectively utilize digital data and support decision-making, current systems make it difficult to do this quickly and accurately. Furthermore, in workplaces where responses must take into account the emotional state of employees, there is a lack of means to grasp employees' stress and anxiety, leading to reduced production efficiency and safety risks. A system that can solve these issues is needed.

[0434] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing a user's emotion in real time, means for incorporating the recognized emotion data into the analysis result, and means for providing information based on the user's emotional state. This makes it possible not only to quickly digitize paper-based documents and perform data analysis and visualization, but also to grasp the user's emotional state in real time using an emotion engine and provide appropriate information.

[0435] "Image capture means for capturing images of paper-based documents" refers to devices and techniques used to capture image data of paper documents.

[0436] An "optical character recognition engine" is software or hardware that analyzes character information in image data and converts it into digital text data.

[0437] "Means for formatting converted text data" refers to a process or device for formatting the text data obtained by the OCR engine into a specific format.

[0438] The "communication means for transmitting formatted data to a server" is a technology for securely transmitting formatted digital data to a server via a network.

[0439] "Means for storing and linking received data in a database" refers to a technology that allows the server to store data received in a database and link it to related information.

[0440] The "means for processing stored data based on an analytical algorithm" is a technology for processing information stored in a database based on a specific analytical algorithm.

[0441] "Means for visualizing analysis results" refers to techniques for visually displaying the data obtained through analysis in the form of graphs or charts in an easy-to-understand manner.

[0442] The "means for transmitting the visualized results to the user's terminal and displaying them" refers to a technique for transmitting the visualized data to the user's device and displaying it on that device.

[0443] The "emotion recognition means for recognizing the user's emotions" is a technology used to analyze the user's facial expressions and voice to determine the user's emotional state.

[0444] "Means for incorporating emotional data into analytical results" refers to a process or technique for adding acquired emotional data to analytical results.

[0445] "Means for providing information based on emotional state" refers to a technique for providing appropriate information or advice according to the user's emotional state.

[0446] This invention combines a system that allows small and medium-sized enterprises and sole proprietors to easily digitize paper-based data, analyze and visualize the data in real time, with an emotion engine that recognizes the user's emotions. To specifically implement this system, the following elements are required:

[0447] 1. Image acquisition method

[0448] The user activates the camera on their smart glasses or smartphone and takes a picture of a paper-based document, which captures the data on the paper medium as image data.

[0449] 2. Optical character recognition means

[0450] The captured image is then analyzed through an optical character recognition engine such as Tesseract OCR, which converts the textual information in the image into digital text data. This process also requires pre-processing techniques to improve the accuracy of image identification.

[0451] 3. Data Formatting Methods

[0452] The converted text data is formatted by date, amount, category, etc. This process is automated by a program to ensure data consistency.

[0453] 4. Means of communication

[0454] The formatted data is securely transmitted to the server using the HTTPS protocol, where it is encrypted to ensure the protection of the information.

[0455] 5. Data storage means

[0456] The server stores the received data in a database and associates it with the user's account information, thereby maintaining data integrity.

[0457] 6. Data Analysis Methods

[0458] The stored data is processed through Python analytical algorithms to analyze sales trends, expense distribution, etc. The analysis results are visualized using Matplotlib and D3.js.

[0459] 7. Results display means

[0460] The visualized results are sent to the user's device in JSON format and are notified to the user via push notification or email. The user can view this data through a dedicated application.

[0461] 8. Emotion recognition means

[0462] Using emotion recognition engines such as "Affectiva," the system extracts emotional data from the user's facial expressions and voice. The emotional data is then incorporated into the analysis results, and information and advice is provided based on the user's emotional state.

[0463] Specific examples

[0464] Factory workers take a photo of their daily reports using smart glasses. OCR converts the reports into digital data and sends it to a server. The data is then analyzed and visualized as an efficiency graph created in Python. Emotion recognition technology also monitors employee stress and sends alerts to managers if necessary.

[0465] Prompt Sentence Examples

[0466] "Smart glasses used on factory floors can take photos of paper-based reports, convert them into text using OCR, and then format the data and send it to a server. Then, data analysis and visualization can be performed, and an emotion engine can be used to detect and respond to employee stress."

[0467] This system not only streamlines work processes in factories, but also allows for flexible responses that take into account the emotional state of employees.

[0468] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0469] Step 1:

[0470] A user takes a photo of a paper-based document using the camera on their smart glasses or smartphone. This image capture means captures the paper-based document as digital image data. The input is the paper-based document, and the output is digital image data. Specifically, the user confirms the appropriate position and lighting, and then presses the shutter button to capture the image.

[0471] Step 2:

[0472] Using the image captured by the device, the optical character recognition (OCR) engine "Tesseract OCR" analyzes the characters in the image and converts them into text data. The input is digital image data, and the output is text data. Specifically, the OCR engine analyzes the image, extracts the character information from the image, and converts it into digital text.

[0473] Step 3:

[0474] The terminal formats the converted text data into items such as date, amount, category, etc. The input is the converted text data, and the output is the formatted data. Specifically, the program automatically analyzes the text data and formats it into the specified format.

[0475] Step 4:

[0476] The terminal sends the formatted data to the server using the HTTPS protocol. This communication method ensures that the formatted data is securely sent to the server. The input is the formatted data, and the output is the completion of data transmission to the server. Specifically, the data is encrypted and sent via HTTPS.

[0477] Step 5:

[0478] The server stores the received data in a database and associates it with the user's account information. This data storage means ensures that the data is consistent and placed in the appropriate location. The input is the data sent to the server, and the output is stored in the database. Specifically, the server analyzes the data and associates it with the user's account information before storing it.

[0479] Step 6:

[0480] The server processes the data stored in the database by applying analytical algorithms. This data analysis method analyzes sales trends, expense distribution, and so on. The input is the data stored in the database, and the output is the analysis results. Specifically, a data analysis algorithm using Python processes the data and generates the results.

[0481] Step 7:

[0482] The server visualizes the analysis results and displays them in the form of graphs and charts using "Matplotlib" and "D3.js." This visualization method makes the analysis results visually easy to understand. The input is the analysis results, and the output is visualized data. Specifically, the server retrieves the analysis results and generates graphs and charts for visual display.

[0483] Step 8:

[0484] The server sends the visualized results to the user's device and displays them in a dedicated application. This means of displaying the results allows the user to easily check the analysis results. The input is the visualized data, and the output is the completion of transmission to the user's device. Specifically, the server generates data in JSON format and sends it to the user's device.

[0485] Step 9:

[0486] The device uses an emotion recognition engine such as "Affectiva" to acquire emotional data from the user's facial expressions and voice. This emotion recognition means determines the user's emotional state in real time. The input is the user's facial expressions and voice, and the output is emotional data. Specifically, the device acquires the user's facial expressions and voice through a camera or microphone and analyzes them using an emotion recognition algorithm.

[0487] Step 10:

[0488] The server incorporates the acquired emotional data into the analysis results and provides information based on the user's emotional state. This emotional data integration means enables responses that correspond to the user's emotions. The inputs are emotional data and analysis results, and the output is integrated information. Specifically, the server uses the emotional data to reevaluate the analysis results and provide additional advice or information.

[0489] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0490] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0491] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0492] [Second embodiment]

[0493] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0494] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0495] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0496] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0497] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0498] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0499] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0500] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0501] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0502] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0503] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0504] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0505] The present invention provides a system that allows small and medium-sized enterprises and sole proprietors to easily digitize paper-based data and analyze and visualize the data in real time. This system is implemented as follows.

[0506] 1. Data capture (camera capture of paper-based documents)

[0507] User:

[0508] Users use their smartphone camera to take a picture of a paper-based document (e.g., a sales ledger or expense receipt), after which the application automatically acquires the image and begins processing it.

[0509] Device:

[0510] The captured image is converted into digital text by the device's optical character recognition (OCR) engine, which then formats the text data according to the user's specifications. This formatting process includes categorizing the data by date, amount, category, and other items.

[0511] 2. Data transmission and storage

[0512] Device:

[0513] The formatted data is encrypted and sent securely from the device to the server using the HTTPS protocol.

[0514] server:

[0515] The data received by the server is first decrypted and then stored in a database based on the user's account information. During the storage process, the data is checked for integrity to ensure that the information registered in the database is accurate.

[0516] 3. Data analysis and visualization

[0517] server:

[0518] The stored data is retrieved periodically or upon user request and predefined analytical algorithms are applied to it, for example to analyze sales trends or popular products. This is done by using SQL queries to retrieve data from the database and then processing it with analytical tools such as Python or R.

[0519] server:

[0520] The results of the analysis are generated in a visually understandable format, such as graphs or charts, using graphing libraries such as Matplotlib or D3.js.

[0521] 4. Sending and displaying results

[0522] server:

[0523] The visualized results data is sent to the user's mobile device in JSON format and is notified to the user via push notification or email.

[0524] Device:

[0525] The visualized data received by the device is displayed in a dedicated application, and users can view this data through the application's interface and quickly check the information they need.

[0526] 5. Utilizing the analysis results

[0527] User:

[0528] Users can make business decisions based on visualized data, such as adjusting inventory based on sales trends or analyzing the distribution of expenses to consider cost-cutting measures.

[0529] Specific examples

[0530] For retailers

[0531] The user, a retail store owner, takes a photo of their daily sales ledger with their smartphone camera. The device sends the image to an OCR engine, which converts and formats it into sales data. This data is securely sent to a server, which stores and analyzes it in a database. The analysis results are visualized as a sales trend graph and displayed on the user's smartphone. The owner can then use this data to adjust their inventory management and purchasing plans.

[0532] For sole proprietors

[0533] Self-employed individuals take photos of expense receipts with their smartphone camera, and the device converts the images into text data using an OCR engine. The converted and formatted expense data is sent to a server and stored in a database. The server analyzes the distribution of expenses by category and sends the results, visualized as a pie chart, to the user's device. Self-employed individuals can use this data to review their expenses and devise cost-cutting measures.

[0534] This allows users to quickly extract necessary information from paper-based data and use the present invention as a useful tool for making business decisions.

[0535] The processing flow will be explained below.

[0536] Step 1:

[0537] The user launches the camera app on their smartphone and takes a photo of a paper-based document (e.g., a sales ledger or expense receipt). The user checks the appropriate position and lighting, then presses the shutter button.

[0538] Step 2:

[0539] The device sends the captured image to an optical character recognition (OCR) engine, which extracts the characters in the image as digital text by analyzing their shapes.

[0540] Step 3:

[0541] The device then formats the text data converted by OCR into a specific format, which involves organizing the data based on items such as date, amount, and category, and converting it into the appropriate format.

[0542] Step 4:

[0543] The device encrypts the formatted data and sends it to the server. This communication uses the HTTPS protocol, ensuring secure transmission of data.

[0544] Step 5:

[0545] The server receives the data, decrypts it, associates it with a specific user account, and stores it in a database, where the data is verified for integrity and placement in the correct location.

[0546] Step 6:

[0547] The server retrieves the data stored in the database periodically or upon user request and applies predefined analytical algorithms, which may include, for example, calculating sales trends or expense distributions.

[0548] Step 7:

[0549] The server generates the analysis results in a visually understandable format using graphing libraries such as Matplotlib or D3.js, for example, to create bar charts, pie charts, and line graphs.

[0550] Step 8:

[0551] The server sends the generated visualization data to the user's device in JSON format, and the user is notified via push notification or email.

[0552] Step 9:

[0553] The device analyzes the received visualization data and displays it in a dedicated application, which the user can use to view graphs and charts through the application interface.

[0554] Step 10:

[0555] Users can make business decisions based on the displayed data, such as adjusting inventory based on sales trends or considering cost-cutting measures based on expense distribution.

[0556] Example 1

[0557] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0558] Currently, digitizing paper-based documents and analyzing and visualizing data quickly and accurately is a burden for small and medium-sized enterprises and sole proprietors. In particular, manual data entry and compilation requires time and effort, which delays business decision-making. Furthermore, appropriate measures to ensure data security and integrity are required.

[0559] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0560] In this invention, the server includes an image acquisition means, a means for converting acquired text data into a specified format using an optical character recognition engine, a means for decrypting encrypted data and storing it in a database, a means for applying an analysis algorithm based on the stored data and processing it, a means for visualizing the analysis results as graphs and charts, and a means for transmitting the visualized result data to a user's terminal and displaying it. This enables small and medium-sized enterprises and sole proprietors to quickly and accurately digitize paper-based data and analyze and visualize the data in real time.

[0561] "Paper-based document" refers to any document containing data or information recorded on physical paper.

[0562] "Image capture means" refers to a device or technology for capturing a digital image of a paper-based document.

[0563] An "optical character recognition engine" refers to software or technology that recognizes characters from images and converts them into digital text.

[0564] "Text data" refers to data in which character information is stored electronically.

[0565] "Means for formatting" refers to devices or technologies for arranging acquired text data to conform to a specified format or structure.

[0566] "Communications" refers to the infrastructure and technology used to transmit data to other devices or systems.

[0567] "Encryption" refers to the technology of converting data using a certain algorithm to protect it.

[0568] "Server" means a computing device that stores and processes data and responds to requests from clients.

[0569] A "database" refers to a system for efficiently storing, retrieving, and managing structured data.

[0570] "Analytical algorithm" refers to a set of methods or procedures used to analyze data and extract specific information or trends.

[0571] "Visualization" refers to the technique of representing data in a visual format such as a graph or chart.

[0572] "User's device" refers to a device such as a computer or smartphone used by a user.

[0573] "Means of transmission" refers to the technology or method used to deliver data to a specified destination.

[0574] "Analysis Results" refers to the information and trends obtained by the analysis algorithm.

[0575] The present invention provides a system that allows small and medium-sized enterprises and sole proprietors to easily digitize paper-based data and analyze and visualize the data in real time. This system is implemented as follows.

[0576] Data capture (camera capture of paper-based documents)

[0577] User:

[0578] Users use their smartphone camera to take a photo of a paper document such as a sales ledger or expense receipt, after which a dedicated application automatically acquires the image and moves on to the next step.

[0579] Device:

[0580] The device passes the captured image to the Tesseract OCR engine, which extracts text data from the image. The extracted text data is then formatted by date, amount, category, and other items. For example, it can be formatted as "2023-10-01, Sales, 10,000 yen."

[0581] Data transmission and storage

[0582] Device:

[0583] The formatted data is encrypted using the AES encryption algorithm and sent to the server via the HTTPS protocol.

[0584] server:

[0585] The server decrypts the received data and stores it in the PostgreSQL database based on the user's account information. The data storage process checks for integrity and ensures accurate information is registered in the database.

[0586] Data Analysis and Visualization

[0587] server:

[0588] The server retrieves the stored data periodically or upon user request and applies analytical algorithms such as Pandas or R to it, for example, to analyze sales trends or popular products.

[0589] server:

[0590] The results of the analysis are generated as graphs and charts using visualization libraries such as Matplotlib and D3.js, for example a line graph showing how sales data fluctuates from month to month.

[0591] Sending and displaying results

[0592] server:

[0593] The visualized results data is sent in JSON format to the user's mobile device and notified to the user via push notification or email.

[0594] Device:

[0595] The device displays the received visualized data using a dedicated application, allowing users to check graphs and charts and quickly obtain the information they need.

[0596] Utilizing analysis results

[0597] User:

[0598] Users can make business decisions based on visualized data, such as adjusting inventory based on sales trends or analyzing the distribution of expenses to consider cost-cutting measures.

[0599] Specific examples

[0600] For retailers

[0601] A retail store owner takes a photo of their daily sales ledger using their smartphone camera. The device sends the image to the Tesseract OCR engine, which converts and formats it into sales data. This data is AES encrypted and sent to the server via HTTPS. The server decrypts the data and stores it in a PostgreSQL database. The Pandas library then calculates monthly sales trends and creates a line graph using Matplotlib. The created graph is sent in JSON format to the user's smartphone. The user receives a push notification, checks the graph, and adjusts inventory management and purchasing plans.

[0602] For sole proprietors

[0603] Self-employed individuals take a photo of their expense receipts with their smartphone camera. The device analyzes the image using the Tesseract OCR engine, extracts expense data, and formats it by category. The formatted expense data is AES encrypted and sent to the server via HTTPS. The server decrypts the data and stores it in a PostgreSQL database. The server analyzes the distribution of expenses by category using the R programming language and generates a pie chart using D3.js. The generated pie chart is sent in JSON format to the user's smartphone, and a push notification is sent. The user can then review the pie chart and devise measures to reduce expenses.

[0604] Example prompts for generative AI models

[0605] Prompt statement:

[0606] "I want to take a photo of my sales ledger with my smartphone, process it using OCR, and convert it into digital data for analysis. How can I create a graph that visually shows sales trends?"

[0607] This allows users to quickly extract necessary information from paper-based data and use the present invention as a useful tool for making business decisions.

[0608] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0609] Step 1:

[0610] User:

[0611] Users use their smartphone's camera to take pictures of paper-based documents such as sales ledgers and expense receipts.

[0612] Input: Paper-based documents (physical ledger books and receipts).

[0613] Output: Early digital images.

[0614] What it does: A user opens their smartphone camera, frames a document, and presses the shutter button, and the image is then sent to a dedicated application.

[0615] Step 2:

[0616] Device:

[0617] The device passes the captured image to the Tesseract OCR engine, which extracts text data from the image.

[0618] Input: Early digital images.

[0619] Output: Raw text data.

[0620] How it works: The dedicated application sends the image to the OCR engine, which analyzes the characters in the image and generates text data. Text data in the format "2023-10-01, Sales, 10,000 yen" is output.

[0621] Step 3:

[0622] Device:

[0623] Formats the text data obtained from the OCR engine into the specified format.

[0624] Input: Raw text data.

[0625] Output: Formatted text data.

[0626] What it does: The algorithm parses the text data and separates it into fields such as date, amount, category, etc. The formatted data is then formatted as "YYYY-MM-DD, category, amount".

[0627] Step 4:

[0628] Device:

[0629] The formatted data is encrypted using the AES encryption algorithm.

[0630] Input: Formatted text data.

[0631] Output: The encrypted data.

[0632] What happens: The application invokes the AES encryption algorithm to encrypt the formatted data, which is then ready to be sent to the server via the HTTPS protocol.

[0633] Step 5:

[0634] Device:

[0635] The encrypted data is sent to the server via the HTTPS protocol.

[0636] Input: Encrypted data.

[0637] Output: The encrypted data sent.

[0638] What happens: Your device sends encrypted data to the server using the HTTPS protocol, where it arrives securely.

[0639] Step 6:

[0640] server:

[0641] The server decrypts the received data using AES encryption.

[0642] Input: The encrypted data sent.

[0643] Output: The decrypted data.

[0644] What happens: The server uses the AES decryption algorithm to decrypt the data and return it to the original text, ready for database entry.

[0645] Step 7:

[0646] server:

[0647] The decrypted data is stored in a database based on the user's account information.

[0648] Input: Decrypted data.

[0649] Output: Data stored in a database.

[0650] What happens: The server establishes a database connection, associates the decrypted data with the corresponding user account, and stores it. The data is checked for integrity when it is stored.

[0651] Step 8:

[0652] server:

[0653] The stored data is retrieved periodically or upon user request and analytical algorithms are applied.

[0654] Input: Data stored in a database.

[0655] Output: Analysis results.

[0656] What it does: The server retrieves data using SQL queries and processes it using tools like Pandas and R. For example, it applies an algorithm to analyze sales trends and calculates the resulting increase or decrease in sales.

[0657] Step 9:

[0658] server:

[0659] The analysis results are generated as graphs and charts using visualization libraries such as Matplotlib and D3.js.

[0660] Input: Analysis results.

[0661] Output: Visualized data (graphs and charts).

[0662] What happens: The server calls a visualization library to create, for example, a line graph showing monthly sales trends. The visualization is generated and ready to send to the user.

[0663] Step 10:

[0664] server:

[0665] The visualized result data is sent in JSON format to the user's mobile device and the user is notified via push notification or email.

[0666] Input: The visualized data.

[0667] Output: The submitted visualization data.

[0668] Specific operation: The server converts the data into JSON format and sends the result to the user via push notification or email. The user's device receives the data.

[0669] Step 11:

[0670] Device:

[0671] The terminal displays the received visualization data using a dedicated application, allowing the user to check graphs and charts.

[0672] Input: The submitted visualization data.

[0673] Output: The displayed graph or chart.

[0674] How it works: A dedicated application analyzes the data and provides visualized information to the user, allowing the user to quickly check the information they need through the application.

[0675] These steps enable users to quickly and accurately digitize paper-based documents and analyze and visualize data in real time.

[0676] (Application example 1)

[0677] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0678] Currently, small and medium-sized enterprises and sole proprietors face many challenges in quickly converting paper-based sales slips and expense receipts into digital data and using that data to make business decisions. Specifically, manual data entry requires time and effort and is prone to human error. Furthermore, real-time data analysis is difficult, making it difficult to immediately identify sales trends and wasteful expenses. This delays the optimization of inventory management and purchasing plans, as well as the development of cost-cutting measures. Therefore, there is a need for a system that can easily digitize this paper-based data and analyze and visualize it in real time.

[0679] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0680] In this invention, the server includes means for sending visualized results to the user's terminal and displaying them, means for processing the stored data based on an analytical algorithm, and means for storing and linking the received data in a database. This makes it possible to convert photos of sales slips and expense receipts taken by the user into digital data and visualize the analysis results, including sales trends and popular products, in real time.

[0681] "Paper-based documents" are records of information written on paper, such as sales slips and expense receipts.

[0682] The "image acquisition means" is a means for taking a photograph of a paper-based document using a camera such as a smartphone and acquiring the image.

[0683] An "optical character recognition engine" is software that extracts text from captured images and converts it into digital data.

[0684] The "means for formatting" is a means for classifying the converted text data into date, amount, category, etc., and formatting it into a predetermined format.

[0685] "Communication means" refers to a means for transmitting formatted data to a server via the Internet.

[0686] A "database" is a collection of digital data stored on a server, where each piece of data is appropriately tagged and linked.

[0687] An "analysis algorithm" is a calculation procedure used to analyze sales trends and popular products based on stored data.

[0688] "Means for visualizing analytical results" refers to means for visually representing the results obtained by analytical algorithms as graphs or charts.

[0689] The "user terminal" is a device such as a smartphone or tablet, which is used to display visualized analysis results.

[0690] "Sales trend" is an indicator that shows fluctuations in sales data over a certain period of time.

[0691] "Popular products" is an indicator of products with high sales during a specific period.

[0692] "Real-time" refers to the user instantly capturing and analyzing ongoing data.

[0693] This invention is a system that enables small and medium-sized enterprises and sole proprietors to easily digitize paper-based sales slips and expense receipts, and then analyze and visualize the data in real time. This system is composed of an image acquisition means, an optical character recognition engine, a data formatting means, a communication means, an analysis algorithm, and a means for visualizing the analysis results.

[0694] Hardware and software used

[0695] 1. Smartphone (device)

[0696] Image acquisition method:

[0697] Users use their smartphone to take a photo of a paper sales slip or expense receipt, and the image is captured directly within the application.

[0698] Optical Character Recognition Engine (OCR):

[0699] On the smartphone, the acquired image is converted into text data using an OCR engine (e.g., Tesseract).

[0700] 2. Server

[0701] Data formatting methods:

[0702] The converted text data is formatted by items such as date, amount, category, etc. before being sent to the server. The formatting process is carried out using a Python script on the smartphone.

[0703] Communication Method:

[0704] The formatted data is sent securely to the server using the HTTPS protocol.

[0705] Storage and linking methods:

[0706] The data received by the server is stored in a database and linked to the user's identification information. This database is typically MySQL.

[0707] Analysis algorithm:

[0708] Algorithms are applied to the data stored in the database to analyze sales trends and popular products, using Python or R.

[0709] Analysis results visualization methods:

[0710] The analyzed results are visualized using graphing libraries such as Matplotlib and D3.js.

[0711] 3. User's Device

[0712] Viewing Results:

[0713] The resulting data is analyzed and visualized on the server and sent to the user's smartphone in JSON format, where it is displayed to the user through a dedicated application.

[0714] Specific examples

[0715] 1. For retailers:

[0716] Retail store owners take photos of their daily sales slips with their smartphone camera. The images are converted into text data using an OCR engine, and then formatted into fields such as date, amount, and category. This data is securely sent to a server for storage and analysis. The analysis results are visualized as sales trends and can be viewed in real time on the user's smartphone.

[0717] 2. For sole proprietors:

[0718] Self-employed individuals take photos of expense receipts with their smartphone camera, and the device converts the images into text data using an OCR engine. This data is then formatted and sent to a server. The analysis results on the server are visualized as expense distribution by category and displayed on the user's smartphone. Self-employed individuals can use this data to plan cost-cutting measures.

[0719] Prompt Sentence Examples

[0720] You can enter a generative AI model using a prompt like this:

[0721] Please generate a concrete example application based on the following invention. The invention is a system that allows small and medium-sized enterprises and sole proprietors to easily digitize paper-based data and analyze and visualize the data in real time. This application is installed on a smartphone and takes photos of sales slips, expense receipts, etc., digitizes them, and then analyzes and visualizes the data based on that.

[0722] Although the embodiments for carrying out the present invention have been described in detail above, the scope of the present invention is not limited to the above examples.

[0723] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0724] Step 1:

[0725] A user takes a photo of a paper-based sales slip or expense receipt with their smartphone camera. This action generates image data. The input is a paper-based document, and the output is image data. Specifically, the user launches the smartphone's camera app and takes a photo of the document.

[0726] Step 2:

[0727] Once the captured image data is saved on the device, the device uses its internal optical character recognition engine (OCR) to convert this image data into text data. The input is image data, and the output is text data. Specifically, the OCR engine (e.g., Tesseract) recognizes the characters in the image and extracts them as text.

[0728] Step 3:

[0729] The converted text data is formatted by a program on the terminal based on the date, amount, category, etc. The input is the converted text data, and the output is the formatted data. Specifically, the formatting algorithm analyzes the text data and separates it into fields.

[0730] Step 4:

[0731] The formatted data is sent to the server in encrypted form using the HTTPS protocol. The input is the formatted data, and the output is the encrypted data transmission completion. Specifically, the terminal opens a secure communication channel and sends the data to the endpoint.

[0732] Step 5:

[0733] The data received by the server is first decrypted, linked to the user's account information, and then stored in a database. The input is encrypted data, and the output is digital data stored in the database. Specifically, the server decodes the data, links it to the corresponding user information, and stores it in the database.

[0734] Step 6:

[0735] The stored data is processed based on analytical algorithms on the server. For example, sales trends and popular products are analyzed. The input is digital data stored in a database, and the output is the analysis results. Specifically, Python or R scripts use SQL queries to retrieve the data and perform the analysis.

[0736] Step 7:

[0737] The analysis results are generated as graphs and charts using a visualization library (e.g., Matplotlib or D3.js). The input is the analysis results, and the output is the visualized data. Specifically, various diagrams and charts are generated based on the analysis results.

[0738] Step 8:

[0739] The visualized result data is sent from the server to the user's device in JSON format. The input is the visualized data, and the output is the completion of sending the JSON format data. Specifically, the server sends the data to the user's device via push notification or email.

[0740] Step 9:

[0741] The data received by the user's device is displayed by a dedicated application. The input is JSON formatted data, and the output is visualized data displayed on the user interface. Specifically, the application parses the JSON data and displays it in an appropriate format.

[0742] Step 10:

[0743] Based on this visualized data, users make business decisions such as inventory management, adjusting purchasing plans, and planning cost reduction measures. The input is the visualized data, and the output is the user's decision. Specifically, users check the data on the application and take action as needed.

[0744] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0745] The present invention combines a system that allows small and medium-sized enterprises and sole proprietors to easily digitize paper-based data, analyze and visualize the data in real time, and further combines it with an emotion engine that recognizes the user's emotions. This system is implemented as follows.

[0746] 1. Data capture (camera capture of paper-based documents)

[0747] User:

[0748] The user launches the camera app on their smartphone, takes a photo of a paper-based document (e.g., a sales ledger or expense receipt), checks the appropriate position and lighting, and presses the shutter button.

[0749] Device:

[0750] The captured image is converted into digital text by the device's optical character recognition (OCR) engine, which then formats the text into a specified format. This formatting process includes sorting the data by date, amount, category, and other items.

[0751] 2. Data transmission and storage

[0752] Device:

[0753] The formatted data is encrypted and sent securely from the device to the server using the HTTPS protocol.

[0754] server:

[0755] Once the data arrives at the server, it is first decrypted and then stored in a database based on the user's account information. During the storage process, the data is verified for integrity and placement in the correct location.

[0756] 3. Data analysis and visualization

[0757] server:

[0758] Data stored in a database is retrieved periodically or upon user request and predefined analytical algorithms are applied, for example, to analyze sales trends or expense distribution.

[0759] server:

[0760] The analysis results are generated in a visually friendly format, using graphing libraries such as Matplotlib and D3.js to create bar charts, pie charts, line graphs, etc.

[0761] 4. Sending and displaying results

[0762] server:

[0763] The generated visualization data is sent to the user's device in JSON format, and the user is notified via push notification or email.

[0764] Device:

[0765] The received visualization data is displayed in a dedicated application, and users can view this data through the application interface and quickly check the information they need.

[0766] 5. Use of Emotion Engine

[0767] Device:

[0768] The emotion engine recognizes the user's facial expressions and voice, and acquires the user's emotion data in real time. The emotion engine uses machine learning algorithms to identify emotions (e.g., joy, anger, sadness, surprise).

[0769] server:

[0770] The acquired emotional data is incorporated into the analysis results, and the information provided takes into account the user's emotional state. For example, if the user expresses dissatisfaction, the system will provide a detailed explanation or additional advice.

[0771] Specific examples

[0772] For retailers

[0773] The user, a retail store owner, takes a photo of their daily sales ledger with their smartphone camera. The device sends the image to an OCR engine, which converts and formats it into sales data. This data is securely sent to a server, where it is stored and analyzed in a database. The analysis results are visualized as a sales trend graph and displayed on the user's smartphone. The emotion engine analyzes the owner's facial expressions, and if a positive emotion is recognized, the system shares success stories and provides further advice.

[0774] For sole proprietors

[0775] Self-employed individuals take photos of expense receipts with their smartphone camera, and the device converts and formats the images into text data using an OCR engine. The converted and formatted expense data is sent to a server and stored in a database. The server analyzes the distribution of expenses by category and sends the results, visualized as a pie chart, to the user's device. If the emotion engine recognizes the user's stress or anxiety, content offering specific ways to reduce costs and mitigation measures is displayed.

[0776] This allows users to receive not just data analysis but also emotionally-based advice, helping them make more effective business decisions.

[0777] The processing flow will be explained below.

[0778] Step 1:

[0779] The user launches the camera app on their smartphone and takes a photo of a paper-based document (e.g., a sales ledger or expense receipt). The user checks the appropriate position and lighting, then presses the shutter button.

[0780] Step 2:

[0781] The device sends the captured image to an optical character recognition (OCR) engine, which extracts the characters in the image as digital text. The OCR engine analyzes the characters in the image and converts the data into text.

[0782] Step 3:

[0783] The device then formats the text data converted by OCR into a specific format, which involves sorting the data by date, amount, category, etc., and converting it into the appropriate format.

[0784] Step 4:

[0785] The device encrypts the formatted data and sends it securely to the server, using the HTTPS protocol to ensure secure data transmission.

[0786] Step 5:

[0787] The server receives the data, decrypts it, associates it with a specific user account, and stores it in a database, where the data is verified for integrity and placement in the correct location.

[0788] Step 6:

[0789] The device transmits the user's facial expressions and voice to the emotion engine, which uses machine learning algorithms to identify the user's emotions (e.g., joy, anger, sadness, surprise) in real time.

[0790] Step 7:

[0791] The server retrieves business data stored in the database periodically or upon user request and applies predefined analytical algorithms to it, for example, analyzing sales trends or expense distribution.

[0792] Step 8:

[0793] The server integrates the analysis results with the user's emotional data and generates the analysis results in a visually easy-to-understand format, such as a bar graph, pie chart, or line graph.

[0794] Step 9:

[0795] The server sends the generated visualization data to the user's device in JSON format, and the user is notified via push notification or email.

[0796] Step 10:

[0797] The device analyzes the received visualization data and displays it in a dedicated application. Users can view graphs and charts through the application interface and quickly obtain the information they need.

[0798] Step 11:

[0799] The server dynamically adjusts the display content based on the user's emotional data. For example, if the user is feeling anxious, the analysis results will include detailed explanations and advice.

[0800] Step 12:

[0801] Users can make business decisions based on the displayed data and additional advice. For example, they can manage inventory based on sales trends or consider cost-cutting measures based on expense distribution. The system also includes advice based on emotional data, allowing users to make effective decisions while reducing psychological burden.

[0802] Example 2

[0803] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0804] For small and medium-sized enterprises and sole proprietors, digitizing paper-based data and analyzing and visualizing it in real time is a time-consuming and labor-intensive task. Furthermore, conventional systems lack the functionality to provide personalized advice that takes into account the user's emotions. As a result, users are limited to simply visualizing the data, making it difficult for them to make effective, emotion-based decisions.

[0805] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an image acquisition means for capturing a paper-based document, a means for converting the captured image into text data using an optical character recognition engine, a means for formatting the converted text data into a specific format, a communication means for transmitting the formatted data to a remote server, a means for storing the received data in a database and associating it with user information, a means for processing the stored data based on an analysis algorithm, a means for visually displaying the analysis results, a means for transmitting the visualized results to the user's device and displaying them, a means for using an emotion recognition engine for analyzing the user's emotions, and a means for including emotion data in the analysis results and providing information tailored to the user. This allows the user to easily digitize paper-based data, analyze and visualize the data in real time, and receive personalized advice based on their emotions.

[0806] "Paper-based documents" refers to text or data that is written on physical paper.

[0807] "Image capture means" refers to a device or method for capturing a paper-based document as a digital image.

[0808] "Optical character recognition engine" refers to software or algorithms that analyze character information from image data and convert it into digital text.

[0809] "Formatting means" refers to processing means that rearranges acquired digital text into a particular format or structure to facilitate storage in a database.

[0810] "Communication means" refers to the network protocols and devices used to transmit data to a remote server.

[0811] "Server" refers to a computer system for storing, processing, and managing data.

[0812] "Database" refers to a data storage system for efficiently storing, retrieving, and managing structured data.

[0813] An "analytical algorithm" refers to a set of computational procedures or rules for processing stored data and extracting useful information.

[0814] "Visualization means" refers to tools and methods for displaying extracted information in an easy-to-understand format, such as graphs or charts.

[0815] "User device" refers to a device (e.g., a smartphone or tablet) that a user owns and uses to display and interact with information.

[0816] An "emotion recognition engine" refers to software or algorithms that analyze and identify emotions from a user's facial expressions and voice.

[0817] "Emotion data" refers to data that indicates the emotional state of the user analyzed by an emotion recognition engine.

[0818] "Information provision" refers to the act of presenting necessary information or advice to a user.

[0819] This invention combines a system that allows small and medium-sized enterprises and individual business owners to easily digitize paper-based data, analyze and visualize the data in real time, and an emotion engine that recognizes user emotions. This system is realized by a specific combination of hardware and software.

[0820] Data capture (camera capture of paper-based documents)

[0821] User:

[0822] A user starts a camera app on a smartphone and takes a picture of a paper-based document (e.g., a sales ledger or expense receipt). For example, the user takes a picture of the sales ledger to obtain a digital image.

[0823] Device:

[0824] The device receives the captured image data and converts it into digital text using an optical character recognition engine (e.g., Google Cloud Vision or Tesseract OCR), then uses the Python Pandas library to format the text data into a specific format (date, amount, category, etc.).

[0825] Data transmission and storage

[0826] Device:

[0827] The formatted data is encrypted and sent to the server using the HTTPS protocol. The SSL / TLS protocol is used for encryption. For example, dates, sales data, etc. are sent securely.

[0828] server:

[0829] The received data is decrypted and stored in a database (e.g., PostgreSQL) based on the user's account information using a web framework such as Django or Flask. The data is verified for consistency and placement in the appropriate fields.

[0830] Data Analysis and Visualization

[0831] server:

[0832] The stored data is retrieved periodically or upon user request and predefined analytical algorithms are applied using machine learning libraries such as Scikit-learn and TensorFlow, for example to analyze sales trends or expense distribution.

[0833] server:

[0834] The analysis results are generated in visually easy-to-understand formats (bar graphs, pie charts, line graphs) using graphing libraries such as Matplotlib and D3.js.

[0835] Sending and displaying results

[0836] server:

[0837] The generated visualization data is sent to the user's device in JSON format, and a message such as "Monthly sales report has been generated" is provided via push notification or email.

[0838] Device:

[0839] The received data is displayed in a dedicated application using frameworks such as React Native or Flutter, and users can view graphs and charts through the app interface.

[0840] Using the Emotion Engine

[0841] Device:

[0842] The system captures the user's facial expressions and voice in real time using Microsoft Azure's Face API and Google Cloud Speech-to-Text API, allowing it to analyze the user's emotions (e.g., joy, anger, sadness, surprise).

[0843] server:

[0844] The acquired emotional data is then incorporated into the analysis results to provide personalized advice and information based on the user's emotional state. For example, the system may provide a message such as, "You seem pleased with your recent sales growth. Would you like to share your success story?"

[0845] Specific examples

[0846] For retailers

[0847] User:

[0848] A retail store owner takes a photo of their daily sales ledger using their smartphone camera.

[0849] Device:

[0850] The device sends the image to the OCR engine, which converts and formats it into sales data.

[0851] server:

[0852] The data stored in the database is analyzed and visualized as a sales trend graph. The visualization results are sent to the user's smartphone and displayed in a dedicated application.

[0853] Emotion Engine:

[0854] The emotion engine analyzes the user's facial expressions, and if a positive emotion is recognized, it will share success stories and provide further advice.

[0855] For sole proprietors

[0856] User:

[0857] A self-employed person takes a photo of an expense receipt with their smartphone camera.

[0858] Device:

[0859] The device converts and formats the image into text data using an OCR engine.

[0860] server:

[0861] The data stored in the database is analyzed, and the distribution of expenses by category is visualized as a pie chart and sent to the user's device.

[0862] Emotion Engine:

[0863] The emotion engine recognizes the user's stress and anxiety and displays content offering specific ways to reduce costs and provide relief.

[0864] Prompt Sentence Examples

[0865] By inputting the following prompt sentence into the generative AI model, an explanation of the system design and processing described above can be generated.

[0866] "Design a system that allows small and medium-sized businesses and sole proprietors to easily digitize paper-based data and analyze and visualize the data in real time. This system should also incorporate an emotion engine that recognizes the user's emotions. Explain the system's processing in natural language by following the steps and providing concrete examples."

[0867] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0868] Step 1: Data capture (camera capture of paper-based documents)

[0869] User:

[0870] A user launches the camera app on their smartphone and takes a picture of a paper-based document (e.g., a sales ledger or expense receipt), checks whether the image is clear, and presses the shutter button. The input is the paper-based document, and the output is a digital image.

[0871] Device:

[0872] The device saves the captured image data and launches an OCR engine (such as Google Cloud Vision or Tesseract OCR). The input is a digital image, and the output is text data, which includes the date and amount.

[0873] Step 2: Formatting the text data

[0874] Device:

[0875] The text data obtained by the OCR engine is formatted using Python's Pandas library. The input is raw text data, and the output is formatted data, for example, "Date: 2023 / 10 / 01, Sales: 20,000 yen." Here, the text data is classified into specific items (date, amount, category, etc.).

[0876] Step 3: Sending data

[0877] Device:

[0878] The formatted data is encrypted using AES and sent to the server using the HTTPS protocol. The input is the formatted data and the output is the encrypted data. The SSL / TLS protocol is used for transmission.

[0879] Step 4: Decrypt and save the data

[0880] server:

[0881] Receives encrypted data and first decrypts it. The input is the encrypted data and the output is the decrypted data. Next, using a web framework such as Django or Flask, the data is saved in a database (e.g., PostgreSQL) based on the user's account information. The input is the decrypted data and the output is the data saved in the database.

[0882] Step 5: Acquire and analyze data

[0883] server:

[0884] The stored data is retrieved periodically or upon user request. The input is the data stored in the database, and the output is the retrieved data. Then, using a machine learning library such as Scikit-learn or TensorFlow, a predefined analytical algorithm is applied. For example, to analyze sales trends or expense distribution. The input is the retrieved data, and the output is the analysis results.

[0885] Step 6: Visualize the data

[0886] server:

[0887] The analysis results are generated in a visually easy-to-understand format (bar graph, pie chart, line graph) using graph creation libraries such as Matplotlib and D3.js. The input is the analysis results, and the output is the visualized data.

[0888] Step 7: Send and display visualization data

[0889] server:

[0890] The generated visualization data is sent to the user's device in JSON format. The input is the visualization data, and the output is the transmitted data. A message such as "Monthly sales report has been generated" is provided via push notification or email.

[0891] Device:

[0892] The received data is displayed in a dedicated application using a framework such as React Native or Flutter. The input is the transmitted JSON data, and the output is a graph or chart.

[0893] Step 8: Emotion recognition and advice provision

[0894] Device:

[0895] The user's facial expressions and voice are acquired in real time using Microsoft Azure's Face API and Google Cloud Speech-to-Text API. The input is the user's facial expressions and voice, and the output is emotion data. The emotion data is sent to the server upon request.

[0896] server:

[0897] Receives emotion data and incorporates it into the analysis results. For example, if the user is satisfied, relevant advice is provided. The input is emotion data, and the output is the advice the user applies. The advice is sent to the user's device in JSON format and displayed in a React Native or Flutter application.

[0898] (Application example 2)

[0899] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0900] Currently, many small and medium-sized enterprises and sole proprietors handle a large amount of paper-based documents, and digitizing this data is a time-consuming and labor-intensive process. While there is also a need for analysis and visualization to effectively utilize digital data and support decision-making, current systems make it difficult to do this quickly and accurately. Furthermore, in workplaces where responses must take into account the emotional state of employees, there is a lack of means to grasp employees' stress and anxiety, leading to reduced production efficiency and safety risks. A system that can solve these issues is needed.

[0901] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing a user's emotion in real time, means for incorporating the recognized emotion data into the analysis result, and means for providing information based on the user's emotional state. This makes it possible not only to quickly digitize paper-based documents and perform data analysis and visualization, but also to grasp the user's emotional state in real time using an emotion engine and provide appropriate information.

[0902] "Image capture means for capturing images of paper-based documents" refers to devices and techniques used to capture image data of paper documents.

[0903] An "optical character recognition engine" is software or hardware that analyzes character information in image data and converts it into digital text data.

[0904] "Means for formatting converted text data" refers to a process or device for formatting the text data obtained by the OCR engine into a specific format.

[0905] The "communication means for transmitting formatted data to a server" is a technology for securely transmitting formatted digital data to a server via a network.

[0906] "Means for storing and linking received data in a database" refers to a technology that allows the server to store data received in a database and link it to related information.

[0907] The "means for processing stored data based on an analytical algorithm" is a technology for processing information stored in a database based on a specific analytical algorithm.

[0908] "Means for visualizing analysis results" refers to techniques for visually displaying the data obtained through analysis in the form of graphs or charts in an easy-to-understand manner.

[0909] The "means for transmitting the visualized results to the user's terminal and displaying them" refers to a technique for transmitting the visualized data to the user's device and displaying it on that device.

[0910] The "emotion recognition means for recognizing the user's emotions" is a technology used to analyze the user's facial expressions and voice to determine the user's emotional state.

[0911] "Means for incorporating emotional data into analytical results" refers to a process or technique for adding acquired emotional data to analytical results.

[0912] "Means for providing information based on emotional state" refers to a technique for providing appropriate information or advice according to the user's emotional state.

[0913] This invention combines a system that allows small and medium-sized enterprises and sole proprietors to easily digitize paper-based data, analyze and visualize the data in real time, with an emotion engine that recognizes the user's emotions. To specifically implement this system, the following elements are required:

[0914] 1. Image acquisition method

[0915] The user activates the camera on their smart glasses or smartphone and takes a picture of a paper-based document, which captures the data on the paper medium as image data.

[0916] 2. Optical character recognition means

[0917] The captured image is then analyzed through an optical character recognition engine such as Tesseract OCR, which converts the textual information in the image into digital text data. This process also requires pre-processing techniques to improve the accuracy of image identification.

[0918] 3. Data Formatting Methods

[0919] The converted text data is formatted by date, amount, category, etc. This process is automated by a program to ensure data consistency.

[0920] 4. Means of communication

[0921] The formatted data is securely transmitted to the server using the HTTPS protocol, where it is encrypted to ensure the protection of the information.

[0922] 5. Data storage means

[0923] The server stores the received data in a database and associates it with the user's account information, thereby maintaining data integrity.

[0924] 6. Data Analysis Methods

[0925] The stored data is processed through Python analytical algorithms to analyze sales trends, expense distribution, etc. The analysis results are visualized using Matplotlib and D3.js.

[0926] 7. Results display means

[0927] The visualized results are sent to the user's device in JSON format and are notified to the user via push notification or email. The user can view this data through a dedicated application.

[0928] 8. Emotion recognition means

[0929] Using emotion recognition engines such as "Affectiva," the system extracts emotional data from the user's facial expressions and voice. The emotional data is then incorporated into the analysis results, and information and advice is provided based on the user's emotional state.

[0930] Specific examples

[0931] Factory workers take a photo of their daily reports using smart glasses. OCR converts the reports into digital data and sends it to a server. The data is then analyzed and visualized as an efficiency graph created in Python. Emotion recognition technology also monitors employee stress and sends alerts to managers if necessary.

[0932] Prompt Sentence Examples

[0933] "Smart glasses used on factory floors can take photos of paper-based reports, convert them into text using OCR, and then format the data and send it to a server. Then, data analysis and visualization can be performed, and an emotion engine can be used to detect and respond to employee stress."

[0934] This system not only streamlines work processes in factories, but also allows for flexible responses that take into account the emotional state of employees.

[0935] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0936] Step 1:

[0937] A user takes a photo of a paper-based document using the camera on their smart glasses or smartphone. This image capture means captures the paper-based document as digital image data. The input is the paper-based document, and the output is digital image data. Specifically, the user confirms the appropriate position and lighting, and then presses the shutter button to capture the image.

[0938] Step 2:

[0939] Using the image captured by the device, the optical character recognition (OCR) engine "Tesseract OCR" analyzes the characters in the image and converts them into text data. The input is digital image data, and the output is text data. Specifically, the OCR engine analyzes the image, extracts the character information from the image, and converts it into digital text.

[0940] Step 3:

[0941] The terminal formats the converted text data into items such as date, amount, category, etc. The input is the converted text data, and the output is the formatted data. Specifically, the program automatically analyzes the text data and formats it into the specified format.

[0942] Step 4:

[0943] The terminal sends the formatted data to the server using the HTTPS protocol. This communication method ensures that the formatted data is securely sent to the server. The input is the formatted data, and the output is the completion of data transmission to the server. Specifically, the data is encrypted and sent via HTTPS.

[0944] Step 5:

[0945] The server stores the received data in a database and associates it with the user's account information. This data storage means ensures that the data is consistent and placed in the appropriate location. The input is the data sent to the server, and the output is stored in the database. Specifically, the server analyzes the data and associates it with the user's account information before storing it.

[0946] Step 6:

[0947] The server processes the data stored in the database by applying analytical algorithms. This data analysis method analyzes sales trends, expense distribution, and so on. The input is the data stored in the database, and the output is the analysis results. Specifically, a data analysis algorithm using Python processes the data and generates the results.

[0948] Step 7:

[0949] The server visualizes the analysis results and displays them in the form of graphs and charts using "Matplotlib" and "D3.js." This visualization method makes the analysis results visually easy to understand. The input is the analysis results, and the output is visualized data. Specifically, the server retrieves the analysis results and generates graphs and charts for visual display.

[0950] Step 8:

[0951] The server sends the visualized results to the user's device and displays them in a dedicated application. This means of displaying the results allows the user to easily check the analysis results. The input is the visualized data, and the output is the completion of transmission to the user's device. Specifically, the server generates data in JSON format and sends it to the user's device.

[0952] Step 9:

[0953] The device uses an emotion recognition engine such as "Affectiva" to acquire emotional data from the user's facial expressions and voice. This emotion recognition means determines the user's emotional state in real time. The input is the user's facial expressions and voice, and the output is emotional data. Specifically, the device acquires the user's facial expressions and voice through a camera or microphone and analyzes them using an emotion recognition algorithm.

[0954] Step 10:

[0955] The server incorporates the acquired emotional data into the analysis results and provides information based on the user's emotional state. This emotional data integration means enables responses that correspond to the user's emotions. The inputs are emotional data and analysis results, and the output is integrated information. Specifically, the server uses the emotional data to reevaluate the analysis results and provide additional advice or information.

[0956] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0957] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0958] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0959] [Third embodiment]

[0960] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0961] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0962] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0963] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0964] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0965] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0966] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0967] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0968] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0969] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0970] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0971] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0972] The present invention provides a system that allows small and medium-sized enterprises and sole proprietors to easily digitize paper-based data and analyze and visualize the data in real time. This system is implemented as follows.

[0973] 1. Data capture (camera capture of paper-based documents)

[0974] User:

[0975] Users use their smartphone camera to take a picture of a paper-based document (e.g., a sales ledger or expense receipt), after which the application automatically acquires the image and begins processing it.

[0976] Device:

[0977] The captured image is converted into digital text by the device's optical character recognition (OCR) engine, which then formats the text data according to the user's specifications. This formatting process includes categorizing the data by date, amount, category, and other items.

[0978] 2. Data transmission and storage

[0979] Device:

[0980] The formatted data is encrypted and sent securely from the device to the server using the HTTPS protocol.

[0981] server:

[0982] The data received by the server is first decrypted and then stored in a database based on the user's account information. During the storage process, the data is checked for integrity to ensure that the information registered in the database is accurate.

[0983] 3. Data analysis and visualization

[0984] server:

[0985] The stored data is retrieved periodically or upon user request and predefined analytical algorithms are applied to it, for example to analyze sales trends or popular products. This is done by using SQL queries to retrieve data from the database and then processing it with analytical tools such as Python or R.

[0986] server:

[0987] The results of the analysis are generated in a visually understandable format, such as graphs or charts, using graphing libraries such as Matplotlib or D3.js.

[0988] 4. Sending and displaying results

[0989] server:

[0990] The visualized results data is sent to the user's mobile device in JSON format and is notified to the user via push notification or email.

[0991] Device:

[0992] The visualized data received by the device is displayed in a dedicated application, and users can view this data through the application's interface and quickly check the information they need.

[0993] 5. Utilizing the analysis results

[0994] User:

[0995] Users can make business decisions based on visualized data, such as adjusting inventory based on sales trends or analyzing the distribution of expenses to consider cost-cutting measures.

[0996] Specific examples

[0997] For retailers

[0998] The user, a retail store owner, takes a photo of their daily sales ledger with their smartphone camera. The device sends the image to an OCR engine, which converts and formats it into sales data. This data is securely sent to a server, which stores and analyzes it in a database. The analysis results are visualized as a sales trend graph and displayed on the user's smartphone. The owner can then use this data to adjust their inventory management and purchasing plans.

[0999] For sole proprietors

[1000] Self-employed individuals take photos of expense receipts with their smartphone camera, and the device converts the images into text data using an OCR engine. The converted and formatted expense data is sent to a server and stored in a database. The server analyzes the distribution of expenses by category and sends the results, visualized as a pie chart, to the user's device. Self-employed individuals can use this data to review their expenses and devise cost-cutting measures.

[1001] This allows users to quickly extract necessary information from paper-based data and use the present invention as a useful tool for making business decisions.

[1002] The processing flow will be explained below.

[1003] Step 1:

[1004] The user launches the camera app on their smartphone and takes a photo of a paper-based document (e.g., a sales ledger or expense receipt). The user checks the appropriate position and lighting, then presses the shutter button.

[1005] Step 2:

[1006] The device sends the captured image to an optical character recognition (OCR) engine, which extracts the characters in the image as digital text by analyzing their shapes.

[1007] Step 3:

[1008] The device then formats the text data converted by OCR into a specific format, which involves organizing the data based on items such as date, amount, and category, and converting it into the appropriate format.

[1009] Step 4:

[1010] The device encrypts the formatted data and sends it to the server. This communication uses the HTTPS protocol, ensuring secure transmission of data.

[1011] Step 5:

[1012] The server receives the data, decrypts it, associates it with a specific user account, and stores it in a database, where the data is verified for integrity and placement in the correct location.

[1013] Step 6:

[1014] The server retrieves the data stored in the database periodically or upon user request and applies predefined analytical algorithms, which may include, for example, calculating sales trends or expense distributions.

[1015] Step 7:

[1016] The server generates the analysis results in a visually understandable format using graphing libraries such as Matplotlib or D3.js, for example, to create bar charts, pie charts, and line graphs.

[1017] Step 8:

[1018] The server sends the generated visualization data to the user's device in JSON format, and the user is notified via push notification or email.

[1019] Step 9:

[1020] The device analyzes the received visualization data and displays it in a dedicated application, which the user can use to view graphs and charts through the application interface.

[1021] Step 10:

[1022] Users can make business decisions based on the displayed data, such as adjusting inventory based on sales trends or considering cost-cutting measures based on expense distribution.

[1023] Example 1

[1024] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1025] Currently, digitizing paper-based documents and analyzing and visualizing data quickly and accurately is a burden for small and medium-sized enterprises and sole proprietors. In particular, manual data entry and compilation requires time and effort, which delays business decision-making. Furthermore, appropriate measures to ensure data security and integrity are required.

[1026] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1027] In this invention, the server includes an image acquisition means, a means for converting acquired text data into a specified format using an optical character recognition engine, a means for decrypting encrypted data and storing it in a database, a means for applying an analysis algorithm based on the stored data and processing it, a means for visualizing the analysis results as graphs and charts, and a means for transmitting the visualized result data to a user's terminal and displaying it. This enables small and medium-sized enterprises and sole proprietors to quickly and accurately digitize paper-based data and analyze and visualize the data in real time.

[1028] "Paper-based document" refers to any document containing data or information recorded on physical paper.

[1029] "Image capture means" refers to a device or technology for capturing a digital image of a paper-based document.

[1030] An "optical character recognition engine" refers to software or technology that recognizes characters from images and converts them into digital text.

[1031] "Text data" refers to data in which character information is stored electronically.

[1032] "Means for formatting" refers to devices or technologies for arranging acquired text data to conform to a specified format or structure.

[1033] "Communications" refers to the infrastructure and technology used to transmit data to other devices or systems.

[1034] "Encryption" refers to the technology of converting data using a certain algorithm to protect it.

[1035] "Server" means a computing device that stores and processes data and responds to requests from clients.

[1036] A "database" refers to a system for efficiently storing, retrieving, and managing structured data.

[1037] "Analytical algorithm" refers to a set of methods or procedures used to analyze data and extract specific information or trends.

[1038] "Visualization" refers to the technique of representing data in a visual format such as a graph or chart.

[1039] "User's device" refers to a device such as a computer or smartphone used by a user.

[1040] "Means of transmission" refers to the technology or method used to deliver data to a specified destination.

[1041] "Analysis Results" refers to the information and trends obtained by the analysis algorithm.

[1042] The present invention provides a system that allows small and medium-sized enterprises and sole proprietors to easily digitize paper-based data and analyze and visualize the data in real time. This system is implemented as follows.

[1043] Data capture (camera capture of paper-based documents)

[1044] User:

[1045] Users use their smartphone camera to take a photo of a paper document such as a sales ledger or expense receipt, after which a dedicated application automatically acquires the image and moves on to the next step.

[1046] Device:

[1047] The device passes the captured image to the Tesseract OCR engine, which extracts text data from the image. The extracted text data is then formatted by date, amount, category, and other items. For example, it can be formatted as "2023-10-01, Sales, 10,000 yen."

[1048] Data transmission and storage

[1049] Device:

[1050] The formatted data is encrypted using the AES encryption algorithm and sent to the server via the HTTPS protocol.

[1051] server:

[1052] The server decrypts the received data and stores it in the PostgreSQL database based on the user's account information. The data storage process checks for integrity and ensures accurate information is registered in the database.

[1053] Data Analysis and Visualization

[1054] server:

[1055] The server retrieves the stored data periodically or upon user request and applies analytical algorithms such as Pandas or R to it, for example, to analyze sales trends or popular products.

[1056] server:

[1057] The results of the analysis are generated as graphs and charts using visualization libraries such as Matplotlib and D3.js, for example a line graph showing how sales data fluctuates from month to month.

[1058] Sending and displaying results

[1059] server:

[1060] The visualized results data is sent in JSON format to the user's mobile device and notified to the user via push notification or email.

[1061] Device:

[1062] The device displays the received visualized data using a dedicated application, allowing users to check graphs and charts and quickly obtain the information they need.

[1063] Utilizing analysis results

[1064] User:

[1065] Users can make business decisions based on visualized data, such as adjusting inventory based on sales trends or analyzing the distribution of expenses to consider cost-cutting measures.

[1066] Specific examples

[1067] For retailers

[1068] A retail store owner takes a photo of their daily sales ledger using their smartphone camera. The device sends the image to the Tesseract OCR engine, which converts and formats it into sales data. This data is AES encrypted and sent to the server via HTTPS. The server decrypts the data and stores it in a PostgreSQL database. The Pandas library then calculates monthly sales trends and creates a line graph using Matplotlib. The created graph is sent in JSON format to the user's smartphone. The user receives a push notification, checks the graph, and adjusts inventory management and purchasing plans.

[1069] For sole proprietors

[1070] Self-employed individuals take a photo of their expense receipts with their smartphone camera. The device analyzes the image using the Tesseract OCR engine, extracts expense data, and formats it by category. The formatted expense data is AES encrypted and sent to the server via HTTPS. The server decrypts the data and stores it in a PostgreSQL database. The server analyzes the distribution of expenses by category using the R programming language and generates a pie chart using D3.js. The generated pie chart is sent in JSON format to the user's smartphone, and a push notification is sent. The user can then review the pie chart and devise measures to reduce expenses.

[1071] Example prompts for generative AI models

[1072] Prompt statement:

[1073] "I want to take a photo of my sales ledger with my smartphone, process it using OCR, and convert it into digital data for analysis. How can I create a graph that visually shows sales trends?"

[1074] This allows users to quickly extract necessary information from paper-based data and use the present invention as a useful tool for making business decisions.

[1075] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1076] Step 1:

[1077] User:

[1078] Users use their smartphone's camera to take pictures of paper-based documents such as sales ledgers and expense receipts.

[1079] Input: Paper-based documents (physical ledger books and receipts).

[1080] Output: Early digital images.

[1081] What it does: A user opens their smartphone camera, frames a document, and presses the shutter button, and the image is then sent to a dedicated application.

[1082] Step 2:

[1083] Device:

[1084] The device passes the captured image to the Tesseract OCR engine, which extracts text data from the image.

[1085] Input: Early digital images.

[1086] Output: Raw text data.

[1087] How it works: The dedicated application sends the image to the OCR engine, which analyzes the characters in the image and generates text data. Text data in the format "2023-10-01, Sales, 10,000 yen" is output.

[1088] Step 3:

[1089] Device:

[1090] Formats the text data obtained from the OCR engine into the specified format.

[1091] Input: Raw text data.

[1092] Output: Formatted text data.

[1093] What it does: The algorithm parses the text data and separates it into fields such as date, amount, category, etc. The formatted data is then formatted as "YYYY-MM-DD, category, amount".

[1094] Step 4:

[1095] Device:

[1096] The formatted data is encrypted using the AES encryption algorithm.

[1097] Input: Formatted text data.

[1098] Output: The encrypted data.

[1099] What happens: The application invokes the AES encryption algorithm to encrypt the formatted data, which is then ready to be sent to the server via the HTTPS protocol.

[1100] Step 5:

[1101] Device:

[1102] The encrypted data is sent to the server via the HTTPS protocol.

[1103] Input: Encrypted data.

[1104] Output: The encrypted data sent.

[1105] What happens: Your device sends encrypted data to the server using the HTTPS protocol, where it arrives securely.

[1106] Step 6:

[1107] server:

[1108] The server decrypts the received data using AES encryption.

[1109] Input: The encrypted data sent.

[1110] Output: The decrypted data.

[1111] What happens: The server uses the AES decryption algorithm to decrypt the data and return it to the original text, ready for database entry.

[1112] Step 7:

[1113] server:

[1114] The decrypted data is stored in a database based on the user's account information.

[1115] Input: Decrypted data.

[1116] Output: Data stored in a database.

[1117] What happens: The server establishes a database connection, associates the decrypted data with the corresponding user account, and stores it. The data is checked for integrity when it is stored.

[1118] Step 8:

[1119] server:

[1120] The stored data is retrieved periodically or upon user request and analytical algorithms are applied.

[1121] Input: Data stored in a database.

[1122] Output: Analysis results.

[1123] What it does: The server retrieves data using SQL queries and processes it using tools like Pandas and R. For example, it applies an algorithm to analyze sales trends and calculates the resulting increase or decrease in sales.

[1124] Step 9:

[1125] server:

[1126] The analysis results are generated as graphs and charts using visualization libraries such as Matplotlib and D3.js.

[1127] Input: Analysis results.

[1128] Output: Visualized data (graphs and charts).

[1129] What happens: The server calls a visualization library to create, for example, a line graph showing monthly sales trends. The visualization is generated and ready to send to the user.

[1130] Step 10:

[1131] server:

[1132] The visualized result data is sent in JSON format to the user's mobile device and the user is notified via push notification or email.

[1133] Input: The visualized data.

[1134] Output: The submitted visualization data.

[1135] Specific operation: The server converts the data into JSON format and sends the result to the user via push notification or email. The user's device receives the data.

[1136] Step 11:

[1137] Device:

[1138] The terminal displays the received visualization data using a dedicated application, allowing the user to check graphs and charts.

[1139] Input: The submitted visualization data.

[1140] Output: The displayed graph or chart.

[1141] How it works: A dedicated application analyzes the data and provides visualized information to the user, allowing the user to quickly check the information they need through the application.

[1142] These steps enable users to quickly and accurately digitize paper-based documents and analyze and visualize data in real time.

[1143] (Application example 1)

[1144] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1145] Currently, small and medium-sized enterprises and sole proprietors face many challenges in quickly converting paper-based sales slips and expense receipts into digital data and using that data to make business decisions. Specifically, manual data entry requires time and effort and is prone to human error. Furthermore, real-time data analysis is difficult, making it difficult to immediately identify sales trends and wasteful expenses. This delays the optimization of inventory management and purchasing plans, as well as the development of cost-cutting measures. Therefore, there is a need for a system that can easily digitize this paper-based data and analyze and visualize it in real time.

[1146] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1147] In this invention, the server includes means for sending visualized results to the user's terminal and displaying them, means for processing the stored data based on an analytical algorithm, and means for storing and linking the received data in a database. This makes it possible to convert photos of sales slips and expense receipts taken by the user into digital data and visualize the analysis results, including sales trends and popular products, in real time.

[1148] "Paper-based documents" are records of information written on paper, such as sales slips and expense receipts.

[1149] The "image acquisition means" is a means for taking a photograph of a paper-based document using a camera such as a smartphone and acquiring the image.

[1150] An "optical character recognition engine" is software that extracts text from captured images and converts it into digital data.

[1151] The "means for formatting" is a means for classifying the converted text data into date, amount, category, etc., and formatting it into a predetermined format.

[1152] "Communication means" refers to a means for transmitting formatted data to a server via the Internet.

[1153] A "database" is a collection of digital data stored on a server, where each piece of data is appropriately tagged and linked.

[1154] An "analysis algorithm" is a calculation procedure used to analyze sales trends and popular products based on stored data.

[1155] "Means for visualizing analytical results" refers to means for visually representing the results obtained by analytical algorithms as graphs or charts.

[1156] The "user terminal" is a device such as a smartphone or tablet, which is used to display visualized analysis results.

[1157] "Sales trend" is an indicator that shows fluctuations in sales data over a certain period of time.

[1158] "Popular products" is an indicator of products with high sales during a specific period.

[1159] "Real-time" refers to the user instantly capturing and analyzing ongoing data.

[1160] This invention is a system that enables small and medium-sized enterprises and sole proprietors to easily digitize paper-based sales slips and expense receipts, and then analyze and visualize the data in real time. This system is composed of an image acquisition means, an optical character recognition engine, a data formatting means, a communication means, an analysis algorithm, and a means for visualizing the analysis results.

[1161] Hardware and software used

[1162] 1. Smartphone (device)

[1163] Image acquisition method:

[1164] Users use their smartphone to take a photo of a paper sales slip or expense receipt, and the image is captured directly within the application.

[1165] Optical Character Recognition Engine (OCR):

[1166] On the smartphone, the acquired image is converted into text data using an OCR engine (e.g., Tesseract).

[1167] 2. Server

[1168] Data formatting methods:

[1169] The converted text data is formatted by items such as date, amount, category, etc. before being sent to the server. The formatting process is carried out using a Python script on the smartphone.

[1170] Communication Method:

[1171] The formatted data is sent securely to the server using the HTTPS protocol.

[1172] Storage and linking methods:

[1173] The data received by the server is stored in a database and linked to the user's identification information. This database is typically MySQL.

[1174] Analysis algorithm:

[1175] Algorithms are applied to the data stored in the database to analyze sales trends and popular products, using Python or R.

[1176] Analysis results visualization methods:

[1177] The analyzed results are visualized using graphing libraries such as Matplotlib and D3.js.

[1178] 3. User's Device

[1179] Viewing Results:

[1180] The resulting data is analyzed and visualized on the server and sent to the user's smartphone in JSON format, where it is displayed to the user through a dedicated application.

[1181] Specific examples

[1182] 1. For retailers:

[1183] Retail store owners take photos of their daily sales slips with their smartphone camera. The images are converted into text data using an OCR engine, and then formatted into fields such as date, amount, and category. This data is securely sent to a server for storage and analysis. The analysis results are visualized as sales trends and can be viewed in real time on the user's smartphone.

[1184] 2. For sole proprietors:

[1185] Self-employed individuals take photos of expense receipts with their smartphone camera, and the device converts the images into text data using an OCR engine. This data is then formatted and sent to a server. The analysis results on the server are visualized as expense distribution by category and displayed on the user's smartphone. Self-employed individuals can use this data to plan cost-cutting measures.

[1186] Prompt Sentence Examples

[1187] You can enter a generative AI model using a prompt like this:

[1188] Please generate a concrete example application based on the following invention. The invention is a system that allows small and medium-sized enterprises and sole proprietors to easily digitize paper-based data and analyze and visualize the data in real time. This application is installed on a smartphone and takes photos of sales slips, expense receipts, etc., digitizes them, and then analyzes and visualizes the data based on that.

[1189] Although the embodiments for carrying out the present invention have been described in detail above, the scope of the present invention is not limited to the above examples.

[1190] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1191] Step 1:

[1192] A user takes a photo of a paper-based sales slip or expense receipt with their smartphone camera. This action generates image data. The input is a paper-based document, and the output is image data. Specifically, the user launches the smartphone's camera app and takes a photo of the document.

[1193] Step 2:

[1194] Once the captured image data is saved on the device, the device uses its internal optical character recognition engine (OCR) to convert this image data into text data. The input is image data, and the output is text data. Specifically, the OCR engine (e.g., Tesseract) recognizes the characters in the image and extracts them as text.

[1195] Step 3:

[1196] The converted text data is formatted by a program on the terminal based on the date, amount, category, etc. The input is the converted text data, and the output is the formatted data. Specifically, the formatting algorithm analyzes the text data and separates it into fields.

[1197] Step 4:

[1198] The formatted data is sent to the server in encrypted form using the HTTPS protocol. The input is the formatted data, and the output is the encrypted data transmission completion. Specifically, the terminal opens a secure communication channel and sends the data to the endpoint.

[1199] Step 5:

[1200] The data received by the server is first decrypted, linked to the user's account information, and then stored in a database. The input is encrypted data, and the output is digital data stored in the database. Specifically, the server decodes the data, links it to the corresponding user information, and stores it in the database.

[1201] Step 6:

[1202] The stored data is processed based on analytical algorithms on the server. For example, sales trends and popular products are analyzed. The input is digital data stored in a database, and the output is the analysis results. Specifically, Python or R scripts use SQL queries to retrieve the data and perform the analysis.

[1203] Step 7:

[1204] The analysis results are generated as graphs and charts using a visualization library (e.g., Matplotlib or D3.js). The input is the analysis results, and the output is the visualized data. Specifically, various diagrams and charts are generated based on the analysis results.

[1205] Step 8:

[1206] The visualized result data is sent from the server to the user's device in JSON format. The input is the visualized data, and the output is the completion of sending the JSON format data. Specifically, the server sends the data to the user's device via push notification or email.

[1207] Step 9:

[1208] The data received by the user's device is displayed by a dedicated application. The input is JSON formatted data, and the output is visualized data displayed on the user interface. Specifically, the application parses the JSON data and displays it in an appropriate format.

[1209] Step 10:

[1210] Based on this visualized data, users make business decisions such as inventory management, adjusting purchasing plans, and planning cost reduction measures. The input is the visualized data, and the output is the user's decision. Specifically, users check the data on the application and take action as needed.

[1211] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1212] The present invention combines a system that allows small and medium-sized enterprises and sole proprietors to easily digitize paper-based data, analyze and visualize the data in real time, and further combines it with an emotion engine that recognizes the user's emotions. This system is implemented as follows.

[1213] 1. Data capture (camera capture of paper-based documents)

[1214] User:

[1215] The user launches the camera app on their smartphone, takes a photo of a paper-based document (e.g., a sales ledger or expense receipt), checks the appropriate position and lighting, and presses the shutter button.

[1216] Device:

[1217] The captured image is converted into digital text by the device's optical character recognition (OCR) engine, which then formats the text into a specified format. This formatting process includes sorting the data by date, amount, category, and other items.

[1218] 2. Data transmission and storage

[1219] Device:

[1220] The formatted data is encrypted and sent securely from the device to the server using the HTTPS protocol.

[1221] server:

[1222] Once the data arrives at the server, it is first decrypted and then stored in a database based on the user's account information. During the storage process, the data is verified for integrity and placement in the correct location.

[1223] 3. Data analysis and visualization

[1224] server:

[1225] Data stored in a database is retrieved periodically or upon user request and predefined analytical algorithms are applied, for example, to analyze sales trends or expense distribution.

[1226] server:

[1227] The analysis results are generated in a visually friendly format, using graphing libraries such as Matplotlib and D3.js to create bar charts, pie charts, line graphs, etc.

[1228] 4. Sending and displaying results

[1229] server:

[1230] The generated visualization data is sent to the user's device in JSON format, and the user is notified via push notification or email.

[1231] Device:

[1232] The received visualization data is displayed in a dedicated application, and users can view this data through the application interface and quickly check the information they need.

[1233] 5. Use of Emotion Engine

[1234] Device:

[1235] The emotion engine recognizes the user's facial expressions and voice, and acquires the user's emotion data in real time. The emotion engine uses machine learning algorithms to identify emotions (e.g., joy, anger, sadness, surprise).

[1236] server:

[1237] The acquired emotional data is incorporated into the analysis results, and the information provided takes into account the user's emotional state. For example, if the user expresses dissatisfaction, the system will provide a detailed explanation or additional advice.

[1238] Specific examples

[1239] For retailers

[1240] The user, a retail store owner, takes a photo of their daily sales ledger with their smartphone camera. The device sends the image to an OCR engine, which converts and formats it into sales data. This data is securely sent to a server, where it is stored and analyzed in a database. The analysis results are visualized as a sales trend graph and displayed on the user's smartphone. The emotion engine analyzes the owner's facial expressions, and if a positive emotion is recognized, the system shares success stories and provides further advice.

[1241] For sole proprietors

[1242] Self-employed individuals take photos of expense receipts with their smartphone camera, and the device converts and formats the images into text data using an OCR engine. The converted and formatted expense data is sent to a server and stored in a database. The server analyzes the distribution of expenses by category and sends the results, visualized as a pie chart, to the user's device. If the emotion engine recognizes the user's stress or anxiety, content offering specific ways to reduce costs and mitigation measures is displayed.

[1243] This allows users to receive not just data analysis but also emotionally-based advice, helping them make more effective business decisions.

[1244] The processing flow will be explained below.

[1245] Step 1:

[1246] The user launches the camera app on their smartphone and takes a photo of a paper-based document (e.g., a sales ledger or expense receipt). The user checks the appropriate position and lighting, then presses the shutter button.

[1247] Step 2:

[1248] The device sends the captured image to an optical character recognition (OCR) engine, which extracts the characters in the image as digital text. The OCR engine analyzes the characters in the image and converts the data into text.

[1249] Step 3:

[1250] The device then formats the text data converted by OCR into a specific format, which involves sorting the data by date, amount, category, etc., and converting it into the appropriate format.

[1251] Step 4:

[1252] The device encrypts the formatted data and sends it securely to the server, using the HTTPS protocol to ensure secure data transmission.

[1253] Step 5:

[1254] The server receives the data, decrypts it, associates it with a specific user account, and stores it in a database, where the data is verified for integrity and placement in the correct location.

[1255] Step 6:

[1256] The device transmits the user's facial expressions and voice to the emotion engine, which uses machine learning algorithms to identify the user's emotions (e.g., joy, anger, sadness, surprise) in real time.

[1257] Step 7:

[1258] The server retrieves business data stored in the database periodically or upon user request and applies predefined analytical algorithms to it, for example, analyzing sales trends or expense distribution.

[1259] Step 8:

[1260] The server integrates the analysis results with the user's emotional data and generates the analysis results in a visually easy-to-understand format, such as a bar graph, pie chart, or line graph.

[1261] Step 9:

[1262] The server sends the generated visualization data to the user's device in JSON format, and the user is notified via push notification or email.

[1263] Step 10:

[1264] The device analyzes the received visualization data and displays it in a dedicated application. Users can view graphs and charts through the application interface and quickly obtain the information they need.

[1265] Step 11:

[1266] The server dynamically adjusts the display content based on the user's emotional data. For example, if the user is feeling anxious, the analysis results will include detailed explanations and advice.

[1267] Step 12:

[1268] Users can make business decisions based on the displayed data and additional advice. For example, they can manage inventory based on sales trends or consider cost-cutting measures based on expense distribution. The system also includes advice based on emotional data, allowing users to make effective decisions while reducing psychological burden.

[1269] Example 2

[1270] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1271] For small and medium-sized enterprises and sole proprietors, digitizing paper-based data and analyzing and visualizing it in real time is a time-consuming and labor-intensive task. Furthermore, conventional systems lack the functionality to provide personalized advice that takes into account the user's emotions. As a result, users are limited to simply visualizing the data, making it difficult for them to make effective, emotion-based decisions.

[1272] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an image acquisition means for capturing a paper-based document, a means for converting the captured image into text data using an optical character recognition engine, a means for formatting the converted text data into a specific format, a communication means for transmitting the formatted data to a remote server, a means for storing the received data in a database and associating it with user information, a means for processing the stored data based on an analysis algorithm, a means for visually displaying the analysis results, a means for transmitting the visualized results to the user's device and displaying them, a means for using an emotion recognition engine for analyzing the user's emotions, and a means for including emotion data in the analysis results and providing information tailored to the user. This allows the user to easily digitize paper-based data, analyze and visualize the data in real time, and receive personalized advice based on their emotions.

[1273] "Paper-based documents" refers to text or data that is written on physical paper.

[1274] "Image capture means" refers to a device or method for capturing a paper-based document as a digital image.

[1275] "Optical character recognition engine" refers to software or algorithms that analyze character information from image data and convert it into digital text.

[1276] "Formatting means" refers to processing means that rearranges acquired digital text into a particular format or structure to facilitate storage in a database.

[1277] "Communication means" refers to the network protocols and devices used to transmit data to a remote server.

[1278] "Server" refers to a computer system for storing, processing, and managing data.

[1279] "Database" refers to a data storage system for efficiently storing, retrieving, and managing structured data.

[1280] An "analytical algorithm" refers to a set of computational procedures or rules for processing stored data and extracting useful information.

[1281] "Visualization means" refers to tools and methods for displaying extracted information in an easy-to-understand format, such as graphs or charts.

[1282] "User device" refers to a device (e.g., a smartphone or tablet) that a user owns and uses to display and interact with information.

[1283] An "emotion recognition engine" refers to software or algorithms that analyze and identify emotions from a user's facial expressions and voice.

[1284] "Emotion data" refers to data that indicates the emotional state of the user analyzed by an emotion recognition engine.

[1285] "Information provision" refers to the act of presenting necessary information or advice to a user.

[1286] This invention combines a system that allows small and medium-sized enterprises and individual business owners to easily digitize paper-based data, analyze and visualize the data in real time, and an emotion engine that recognizes user emotions. This system is realized by a specific combination of hardware and software.

[1287] Data capture (camera capture of paper-based documents)

[1288] User:

[1289] A user starts a camera app on a smartphone and takes a picture of a paper-based document (e.g., a sales ledger or expense receipt). For example, the user takes a picture of the sales ledger to obtain a digital image.

[1290] Device:

[1291] The device receives the captured image data and converts it into digital text using an optical character recognition engine (e.g., Google Cloud Vision or Tesseract OCR), then uses the Python Pandas library to format the text data into a specific format (date, amount, category, etc.).

[1292] Data transmission and storage

[1293] Device:

[1294] The formatted data is encrypted and sent to the server using the HTTPS protocol. The SSL / TLS protocol is used for encryption. For example, dates, sales data, etc. are sent securely.

[1295] server:

[1296] The received data is decrypted and stored in a database (e.g., PostgreSQL) based on the user's account information using a web framework such as Django or Flask. The data is verified for consistency and placement in the appropriate fields.

[1297] Data Analysis and Visualization

[1298] server:

[1299] The stored data is retrieved periodically or upon user request and predefined analytical algorithms are applied using machine learning libraries such as Scikit-learn and TensorFlow, for example to analyze sales trends or expense distribution.

[1300] server:

[1301] The analysis results are generated in visually easy-to-understand formats (bar graphs, pie charts, line graphs) using graphing libraries such as Matplotlib and D3.js.

[1302] Sending and displaying results

[1303] server:

[1304] The generated visualization data is sent to the user's device in JSON format, and a message such as "Monthly sales report has been generated" is provided via push notification or email.

[1305] Device:

[1306] The received data is displayed in a dedicated application using frameworks such as React Native or Flutter, and users can view graphs and charts through the app interface.

[1307] Using the Emotion Engine

[1308] Device:

[1309] The system captures the user's facial expressions and voice in real time using Microsoft Azure's Face API and Google Cloud Speech-to-Text API, allowing it to analyze the user's emotions (e.g., joy, anger, sadness, surprise).

[1310] server:

[1311] The acquired emotional data is then incorporated into the analysis results to provide personalized advice and information based on the user's emotional state. For example, the system may provide a message such as, "You seem pleased with your recent sales growth. Would you like to share your success story?"

[1312] Specific examples

[1313] For retailers

[1314] User:

[1315] A retail store owner takes a photo of their daily sales ledger using their smartphone camera.

[1316] Device:

[1317] The device sends the image to the OCR engine, which converts and formats it into sales data.

[1318] server:

[1319] The data stored in the database is analyzed and visualized as a sales trend graph. The visualization results are sent to the user's smartphone and displayed in a dedicated application.

[1320] Emotion Engine:

[1321] The emotion engine analyzes the user's facial expressions, and if a positive emotion is recognized, it will share success stories and provide further advice.

[1322] For sole proprietors

[1323] User:

[1324] A self-employed person takes a photo of an expense receipt with their smartphone camera.

[1325] Device:

[1326] The device converts and formats the image into text data using an OCR engine.

[1327] server:

[1328] The data stored in the database is analyzed, and the distribution of expenses by category is visualized as a pie chart and sent to the user's device.

[1329] Emotion Engine:

[1330] The emotion engine recognizes the user's stress and anxiety and displays content offering specific ways to reduce costs and provide relief.

[1331] Prompt Sentence Examples

[1332] By inputting the following prompt sentence into the generative AI model, an explanation of the system design and processing described above can be generated.

[1333] "Design a system that allows small and medium-sized businesses and sole proprietors to easily digitize paper-based data and analyze and visualize the data in real time. This system should also incorporate an emotion engine that recognizes the user's emotions. Explain the system's processing in natural language by following the steps and providing concrete examples."

[1334] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1335] Step 1: Data capture (camera capture of paper-based documents)

[1336] User:

[1337] A user launches the camera app on their smartphone and takes a picture of a paper-based document (e.g., a sales ledger or expense receipt), checks whether the image is clear, and presses the shutter button. The input is the paper-based document, and the output is a digital image.

[1338] Device:

[1339] The device saves the captured image data and launches an OCR engine (such as Google Cloud Vision or Tesseract OCR). The input is a digital image, and the output is text data, which includes the date and amount.

[1340] Step 2: Formatting the text data

[1341] Device:

[1342] The text data obtained by the OCR engine is formatted using Python's Pandas library. The input is raw text data, and the output is formatted data, for example, "Date: 2023 / 10 / 01, Sales: 20,000 yen." Here, the text data is classified into specific items (date, amount, category, etc.).

[1343] Step 3: Sending data

[1344] Device:

[1345] The formatted data is encrypted using AES and sent to the server using the HTTPS protocol. The input is the formatted data and the output is the encrypted data. The SSL / TLS protocol is used for transmission.

[1346] Step 4: Decrypt and save the data

[1347] server:

[1348] Receives encrypted data and first decrypts it. The input is the encrypted data and the output is the decrypted data. Next, using a web framework such as Django or Flask, the data is saved in a database (e.g., PostgreSQL) based on the user's account information. The input is the decrypted data and the output is the data saved in the database.

[1349] Step 5: Acquire and analyze data

[1350] server:

[1351] The stored data is retrieved periodically or upon user request. The input is the data stored in the database, and the output is the retrieved data. Then, using a machine learning library such as Scikit-learn or TensorFlow, a predefined analytical algorithm is applied. For example, to analyze sales trends or expense distribution. The input is the retrieved data, and the output is the analysis results.

[1352] Step 6: Visualize the data

[1353] server:

[1354] The analysis results are generated in a visually easy-to-understand format (bar graph, pie chart, line graph) using graph creation libraries such as Matplotlib and D3.js. The input is the analysis results, and the output is the visualized data.

[1355] Step 7: Send and display visualization data

[1356] server:

[1357] The generated visualization data is sent to the user's device in JSON format. The input is the visualization data, and the output is the transmitted data. A message such as "Monthly sales report has been generated" is provided via push notification or email.

[1358] Device:

[1359] The received data is displayed in a dedicated application using a framework such as React Native or Flutter. The input is the transmitted JSON data, and the output is a graph or chart.

[1360] Step 8: Emotion recognition and advice provision

[1361] Device:

[1362] The user's facial expressions and voice are acquired in real time using Microsoft Azure's Face API and Google Cloud Speech-to-Text API. The input is the user's facial expressions and voice, and the output is emotion data. The emotion data is sent to the server upon request.

[1363] server:

[1364] Receives emotion data and incorporates it into the analysis results. For example, if the user is satisfied, relevant advice is provided. The input is emotion data, and the output is the advice the user applies. The advice is sent to the user's device in JSON format and displayed in a React Native or Flutter application.

[1365] (Application example 2)

[1366] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1367] Currently, many small and medium-sized enterprises and sole proprietors handle a large amount of paper-based documents, and digitizing this data is a time-consuming and labor-intensive process. While there is also a need for analysis and visualization to effectively utilize digital data and support decision-making, current systems make it difficult to do this quickly and accurately. Furthermore, in workplaces where responses must take into account the emotional state of employees, there is a lack of means to grasp employees' stress and anxiety, leading to reduced production efficiency and safety risks. A system that can solve these issues is needed.

[1368] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing a user's emotion in real time, means for incorporating the recognized emotion data into the analysis result, and means for providing information based on the user's emotional state. This makes it possible not only to quickly digitize paper-based documents and perform data analysis and visualization, but also to grasp the user's emotional state in real time using an emotion engine and provide appropriate information.

[1369] "Image capture means for capturing images of paper-based documents" refers to devices and techniques used to capture image data of paper documents.

[1370] An "optical character recognition engine" is software or hardware that analyzes character information in image data and converts it into digital text data.

[1371] "Means for formatting converted text data" refers to a process or device for formatting the text data obtained by the OCR engine into a specific format.

[1372] The "communication means for transmitting formatted data to a server" is a technology for securely transmitting formatted digital data to a server via a network.

[1373] "Means for storing and linking received data in a database" refers to a technology that allows the server to store data received in a database and link it to related information.

[1374] The "means for processing stored data based on an analytical algorithm" is a technology for processing information stored in a database based on a specific analytical algorithm.

[1375] "Means for visualizing analysis results" refers to techniques for visually displaying the data obtained through analysis in the form of graphs or charts in an easy-to-understand manner.

[1376] The "means for transmitting the visualized results to the user's terminal and displaying them" refers to a technique for transmitting the visualized data to the user's device and displaying it on that device.

[1377] The "emotion recognition means for recognizing the user's emotions" is a technology used to analyze the user's facial expressions and voice to determine the user's emotional state.

[1378] "Means for incorporating emotional data into analytical results" refers to a process or technique for adding acquired emotional data to analytical results.

[1379] "Means for providing information based on emotional state" refers to a technique for providing appropriate information or advice according to the user's emotional state.

[1380] This invention combines a system that allows small and medium-sized enterprises and sole proprietors to easily digitize paper-based data, analyze and visualize the data in real time, with an emotion engine that recognizes the user's emotions. To specifically implement this system, the following elements are required:

[1381] 1. Image acquisition method

[1382] The user activates the camera on their smart glasses or smartphone and takes a picture of a paper-based document, which captures the data on the paper medium as image data.

[1383] 2. Optical character recognition means

[1384] The captured image is then analyzed through an optical character recognition engine such as Tesseract OCR, which converts the textual information in the image into digital text data. This process also requires pre-processing techniques to improve the accuracy of image identification.

[1385] 3. Data Formatting Methods

[1386] The converted text data is formatted by date, amount, category, etc. This process is automated by a program to ensure data consistency.

[1387] 4. Means of communication

[1388] The formatted data is securely transmitted to the server using the HTTPS protocol, where it is encrypted to ensure the protection of the information.

[1389] 5. Data storage means

[1390] The server stores the received data in a database and associates it with the user's account information, thereby maintaining data integrity.

[1391] 6. Data Analysis Methods

[1392] The stored data is processed through Python analytical algorithms to analyze sales trends, expense distribution, etc. The analysis results are visualized using Matplotlib and D3.js.

[1393] 7. Results display means

[1394] The visualized results are sent to the user's device in JSON format and are notified to the user via push notification or email. The user can view this data through a dedicated application.

[1395] 8. Emotion recognition means

[1396] Using emotion recognition engines such as "Affectiva," the system extracts emotional data from the user's facial expressions and voice. The emotional data is then incorporated into the analysis results, and information and advice is provided based on the user's emotional state.

[1397] Specific examples

[1398] Factory workers take a photo of their daily reports using smart glasses. OCR converts the reports into digital data and sends it to a server. The data is then analyzed and visualized as an efficiency graph created in Python. Emotion recognition technology also monitors employee stress and sends alerts to managers if necessary.

[1399] Prompt Sentence Examples

[1400] "Smart glasses used on factory floors can take photos of paper-based reports, convert them into text using OCR, and then format the data and send it to a server. Then, data analysis and visualization can be performed, and an emotion engine can be used to detect and respond to employee stress."

[1401] This system not only streamlines work processes in factories, but also allows for flexible responses that take into account the emotional state of employees.

[1402] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1403] Step 1:

[1404] A user takes a photo of a paper-based document using the camera on their smart glasses or smartphone. This image capture means captures the paper-based document as digital image data. The input is the paper-based document, and the output is digital image data. Specifically, the user confirms the appropriate position and lighting, and then presses the shutter button to capture the image.

[1405] Step 2:

[1406] Using the image captured by the device, the optical character recognition (OCR) engine "Tesseract OCR" analyzes the characters in the image and converts them into text data. The input is digital image data, and the output is text data. Specifically, the OCR engine analyzes the image, extracts the character information from the image, and converts it into digital text.

[1407] Step 3:

[1408] The terminal formats the converted text data into items such as date, amount, category, etc. The input is the converted text data, and the output is the formatted data. Specifically, the program automatically analyzes the text data and formats it into the specified format.

[1409] Step 4:

[1410] The terminal sends the formatted data to the server using the HTTPS protocol. This communication method ensures that the formatted data is securely sent to the server. The input is the formatted data, and the output is the completion of data transmission to the server. Specifically, the data is encrypted and sent via HTTPS.

[1411] Step 5:

[1412] The server stores the received data in a database and associates it with the user's account information. This data storage means ensures that the data is consistent and placed in the appropriate location. The input is the data sent to the server, and the output is stored in the database. Specifically, the server analyzes the data and associates it with the user's account information before storing it.

[1413] Step 6:

[1414] The server processes the data stored in the database by applying analytical algorithms. This data analysis method analyzes sales trends, expense distribution, and so on. The input is the data stored in the database, and the output is the analysis results. Specifically, a data analysis algorithm using Python processes the data and generates the results.

[1415] Step 7:

[1416] The server visualizes the analysis results and displays them in the form of graphs and charts using "Matplotlib" and "D3.js." This visualization method makes the analysis results visually easy to understand. The input is the analysis results, and the output is visualized data. Specifically, the server retrieves the analysis results and generates graphs and charts for visual display.

[1417] Step 8:

[1418] The server sends the visualized results to the user's device and displays them in a dedicated application. This means of displaying the results allows the user to easily check the analysis results. The input is the visualized data, and the output is the completion of transmission to the user's device. Specifically, the server generates data in JSON format and sends it to the user's device.

[1419] Step 9:

[1420] The device uses an emotion recognition engine such as "Affectiva" to acquire emotional data from the user's facial expressions and voice. This emotion recognition means determines the user's emotional state in real time. The input is the user's facial expressions and voice, and the output is emotional data. Specifically, the device acquires the user's facial expressions and voice through a camera or microphone and analyzes them using an emotion recognition algorithm.

[1421] Step 10:

[1422] The server incorporates the acquired emotional data into the analysis results and provides information based on the user's emotional state. This emotional data integration means enables responses that correspond to the user's emotions. The inputs are emotional data and analysis results, and the output is integrated information. Specifically, the server uses the emotional data to reevaluate the analysis results and provide additional advice or information.

[1423] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1424] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1425] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1426] [Fourth embodiment]

[1427] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1428] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1429] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1430] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1431] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1432] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1433] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1434] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1435] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1436] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1437] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1438] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1439] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1440] The present invention provides a system that allows small and medium-sized enterprises and sole proprietors to easily digitize paper-based data and analyze and visualize the data in real time. This system is implemented as follows.

[1441] 1. Data capture (camera capture of paper-based documents)

[1442] User:

[1443] Users use their smartphone camera to take a picture of a paper-based document (e.g., a sales ledger or expense receipt), after which the application automatically acquires the image and begins processing it.

[1444] Device:

[1445] The captured image is converted into digital text by the device's optical character recognition (OCR) engine, which then formats the text data according to the user's specifications. This formatting process includes categorizing the data by date, amount, category, and other items.

[1446] 2. Data transmission and storage

[1447] Device:

[1448] The formatted data is encrypted and sent securely from the device to the server using the HTTPS protocol.

[1449] server:

[1450] The data received by the server is first decrypted and then stored in a database based on the user's account information. During the storage process, the data is checked for integrity to ensure that the information registered in the database is accurate.

[1451] 3. Data analysis and visualization

[1452] server:

[1453] The stored data is retrieved periodically or upon user request and predefined analytical algorithms are applied to it, for example to analyze sales trends or popular products. This is done by using SQL queries to retrieve data from the database and then processing it with analytical tools such as Python or R.

[1454] server:

[1455] The results of the analysis are generated in a visually understandable format, such as graphs or charts, using graphing libraries such as Matplotlib or D3.js.

[1456] 4. Sending and displaying results

[1457] server:

[1458] The visualized results data is sent to the user's mobile device in JSON format and is notified to the user via push notification or email.

[1459] Device:

[1460] The visualized data received by the device is displayed in a dedicated application, and users can view this data through the application's interface and quickly check the information they need.

[1461] 5. Utilizing the analysis results

[1462] User:

[1463] Users can make business decisions based on visualized data, such as adjusting inventory based on sales trends or analyzing the distribution of expenses to consider cost-cutting measures.

[1464] Specific examples

[1465] For retailers

[1466] The user, a retail store owner, takes a photo of their daily sales ledger with their smartphone camera. The device sends the image to an OCR engine, which converts and formats it into sales data. This data is securely sent to a server, which stores and analyzes it in a database. The analysis results are visualized as a sales trend graph and displayed on the user's smartphone. The owner can then use this data to adjust their inventory management and purchasing plans.

[1467] For sole proprietors

[1468] Self-employed individuals take photos of expense receipts with their smartphone camera, and the device converts the images into text data using an OCR engine. The converted and formatted expense data is sent to a server and stored in a database. The server analyzes the distribution of expenses by category and sends the results, visualized as a pie chart, to the user's device. Self-employed individuals can use this data to review their expenses and devise cost-cutting measures.

[1469] This allows users to quickly extract necessary information from paper-based data and use the present invention as a useful tool for making business decisions.

[1470] The processing flow will be explained below.

[1471] Step 1:

[1472] The user launches the camera app on their smartphone and takes a photo of a paper-based document (e.g., a sales ledger or expense receipt). The user checks the appropriate position and lighting, then presses the shutter button.

[1473] Step 2:

[1474] The device sends the captured image to an optical character recognition (OCR) engine, which extracts the characters in the image as digital text by analyzing their shapes.

[1475] Step 3:

[1476] The device then formats the text data converted by OCR into a specific format, which involves organizing the data based on items such as date, amount, and category, and converting it into the appropriate format.

[1477] Step 4:

[1478] The device encrypts the formatted data and sends it to the server. This communication uses the HTTPS protocol, ensuring secure transmission of data.

[1479] Step 5:

[1480] The server receives the data, decrypts it, associates it with a specific user account, and stores it in a database, where the data is verified for integrity and placement in the correct location.

[1481] Step 6:

[1482] The server retrieves the data stored in the database periodically or upon user request and applies predefined analytical algorithms, which may include, for example, calculating sales trends or expense distributions.

[1483] Step 7:

[1484] The server generates the analysis results in a visually understandable format using graphing libraries such as Matplotlib or D3.js, for example, to create bar charts, pie charts, and line graphs.

[1485] Step 8:

[1486] The server sends the generated visualization data to the user's device in JSON format, and the user is notified via push notification or email.

[1487] Step 9:

[1488] The device analyzes the received visualization data and displays it in a dedicated application, which the user can use to view graphs and charts through the application interface.

[1489] Step 10:

[1490] Users can make business decisions based on the displayed data, such as adjusting inventory based on sales trends or considering cost-cutting measures based on expense distribution.

[1491] Example 1

[1492] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1493] Currently, digitizing paper-based documents and analyzing and visualizing data quickly and accurately is a burden for small and medium-sized enterprises and sole proprietors. In particular, manual data entry and compilation requires time and effort, which delays business decision-making. Furthermore, appropriate measures to ensure data security and integrity are required.

[1494] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1495] In this invention, the server includes an image acquisition means, a means for converting acquired text data into a specified format using an optical character recognition engine, a means for decrypting encrypted data and storing it in a database, a means for applying an analysis algorithm based on the stored data and processing it, a means for visualizing the analysis results as graphs and charts, and a means for transmitting the visualized result data to a user's terminal and displaying it. This enables small and medium-sized enterprises and sole proprietors to quickly and accurately digitize paper-based data and analyze and visualize the data in real time.

[1496] "Paper-based document" refers to any document containing data or information recorded on physical paper.

[1497] "Image capture means" refers to a device or technology for capturing a digital image of a paper-based document.

[1498] An "optical character recognition engine" refers to software or technology that recognizes characters from images and converts them into digital text.

[1499] "Text data" refers to data in which character information is stored electronically.

[1500] "Means for formatting" refers to devices or technologies for arranging acquired text data to conform to a specified format or structure.

[1501] "Communications" refers to the infrastructure and technology used to transmit data to other devices or systems.

[1502] "Encryption" refers to the technology of converting data using a certain algorithm to protect it.

[1503] "Server" means a computing device that stores and processes data and responds to requests from clients.

[1504] A "database" refers to a system for efficiently storing, retrieving, and managing structured data.

[1505] "Analytical algorithm" refers to a set of methods or procedures used to analyze data and extract specific information or trends.

[1506] "Visualization" refers to the technique of representing data in a visual format such as a graph or chart.

[1507] "User's device" refers to a device such as a computer or smartphone used by a user.

[1508] "Means of transmission" refers to the technology or method used to deliver data to a specified destination.

[1509] "Analysis Results" refers to the information and trends obtained by the analysis algorithm.

[1510] The present invention provides a system that allows small and medium-sized enterprises and sole proprietors to easily digitize paper-based data and analyze and visualize the data in real time. This system is implemented as follows.

[1511] Data capture (camera capture of paper-based documents)

[1512] User:

[1513] Users use their smartphone camera to take a photo of a paper document such as a sales ledger or expense receipt, after which a dedicated application automatically acquires the image and moves on to the next step.

[1514] Device:

[1515] The device passes the captured image to the Tesseract OCR engine, which extracts text data from the image. The extracted text data is then formatted by date, amount, category, and other items. For example, it can be formatted as "2023-10-01, Sales, 10,000 yen."

[1516] Data transmission and storage

[1517] Device:

[1518] The formatted data is encrypted using the AES encryption algorithm and sent to the server via the HTTPS protocol.

[1519] server:

[1520] The server decrypts the received data and stores it in the PostgreSQL database based on the user's account information. The data storage process checks for integrity and ensures accurate information is registered in the database.

[1521] Data Analysis and Visualization

[1522] server:

[1523] The server retrieves the stored data periodically or upon user request and applies analytical algorithms such as Pandas or R to it, for example, to analyze sales trends or popular products.

[1524] server:

[1525] The results of the analysis are generated as graphs and charts using visualization libraries such as Matplotlib and D3.js, for example a line graph showing how sales data fluctuates from month to month.

[1526] Sending and displaying results

[1527] server:

[1528] The visualized results data is sent in JSON format to the user's mobile device and notified to the user via push notification or email.

[1529] Device:

[1530] The device displays the received visualized data using a dedicated application, allowing users to check graphs and charts and quickly obtain the information they need.

[1531] Utilizing analysis results

[1532] User:

[1533] Users can make business decisions based on visualized data, such as adjusting inventory based on sales trends or analyzing the distribution of expenses to consider cost-cutting measures.

[1534] Specific examples

[1535] For retailers

[1536] A retail store owner takes a photo of their daily sales ledger using their smartphone camera. The device sends the image to the Tesseract OCR engine, which converts and formats it into sales data. This data is AES encrypted and sent to the server via HTTPS. The server decrypts the data and stores it in a PostgreSQL database. The Pandas library then calculates monthly sales trends and creates a line graph using Matplotlib. The created graph is sent in JSON format to the user's smartphone. The user receives a push notification, checks the graph, and adjusts inventory management and purchasing plans.

[1537] For sole proprietors

[1538] Self-employed individuals take a photo of their expense receipts with their smartphone camera. The device analyzes the image using the Tesseract OCR engine, extracts expense data, and formats it by category. The formatted expense data is AES encrypted and sent to the server via HTTPS. The server decrypts the data and stores it in a PostgreSQL database. The server analyzes the distribution of expenses by category using the R programming language and generates a pie chart using D3.js. The generated pie chart is sent in JSON format to the user's smartphone, and a push notification is sent. The user can then review the pie chart and devise measures to reduce expenses.

[1539] Example prompts for generative AI models

[1540] Prompt statement:

[1541] "I want to take a photo of my sales ledger with my smartphone, process it using OCR, and convert it into digital data for analysis. How can I create a graph that visually shows sales trends?"

[1542] This allows users to quickly extract necessary information from paper-based data and use the present invention as a useful tool for making business decisions.

[1543] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1544] Step 1:

[1545] User:

[1546] Users use their smartphone's camera to take pictures of paper-based documents such as sales ledgers and expense receipts.

[1547] Input: Paper-based documents (physical ledger books and receipts).

[1548] Output: Early digital images.

[1549] What it does: A user opens their smartphone camera, frames a document, and presses the shutter button, and the image is then sent to a dedicated application.

[1550] Step 2:

[1551] Device:

[1552] The device passes the captured image to the Tesseract OCR engine, which extracts text data from the image.

[1553] Input: Early digital images.

[1554] Output: Raw text data.

[1555] How it works: The dedicated application sends the image to the OCR engine, which analyzes the characters in the image and generates text data. Text data in the format "2023-10-01, Sales, 10,000 yen" is output.

[1556] Step 3:

[1557] Device:

[1558] Formats the text data obtained from the OCR engine into the specified format.

[1559] Input: Raw text data.

[1560] Output: Formatted text data.

[1561] What it does: The algorithm parses the text data and separates it into fields such as date, amount, category, etc. The formatted data is then formatted as "YYYY-MM-DD, category, amount".

[1562] Step 4:

[1563] Device:

[1564] The formatted data is encrypted using the AES encryption algorithm.

[1565] Input: Formatted text data.

[1566] Output: The encrypted data.

[1567] What happens: The application invokes the AES encryption algorithm to encrypt the formatted data, which is then ready to be sent to the server via the HTTPS protocol.

[1568] Step 5:

[1569] Device:

[1570] The encrypted data is sent to the server via the HTTPS protocol.

[1571] Input: Encrypted data.

[1572] Output: The encrypted data sent.

[1573] What happens: Your device sends encrypted data to the server using the HTTPS protocol, where it arrives securely.

[1574] Step 6:

[1575] server:

[1576] The server decrypts the received data using AES encryption.

[1577] Input: The encrypted data sent.

[1578] Output: The decrypted data.

[1579] What happens: The server uses the AES decryption algorithm to decrypt the data and return it to the original text, ready for database entry.

[1580] Step 7:

[1581] server:

[1582] The decrypted data is stored in a database based on the user's account information.

[1583] Input: Decrypted data.

[1584] Output: Data stored in a database.

[1585] What happens: The server establishes a database connection, associates the decrypted data with the corresponding user account, and stores it. The data is checked for integrity when it is stored.

[1586] Step 8:

[1587] server:

[1588] The stored data is retrieved periodically or upon user request and analytical algorithms are applied.

[1589] Input: Data stored in a database.

[1590] Output: Analysis results.

[1591] What it does: The server retrieves data using SQL queries and processes it using tools like Pandas and R. For example, it applies an algorithm to analyze sales trends and calculates the resulting increase or decrease in sales.

[1592] Step 9:

[1593] server:

[1594] The analysis results are generated as graphs and charts using visualization libraries such as Matplotlib and D3.js.

[1595] Input: Analysis results.

[1596] Output: Visualized data (graphs and charts).

[1597] What happens: The server calls a visualization library to create, for example, a line graph showing monthly sales trends. The visualization is generated and ready to send to the user.

[1598] Step 10:

[1599] server:

[1600] The visualized result data is sent in JSON format to the user's mobile device and the user is notified via push notification or email.

[1601] Input: The visualized data.

[1602] Output: The submitted visualization data.

[1603] Specific operation: The server converts the data into JSON format and sends the result to the user via push notification or email. The user's device receives the data.

[1604] Step 11:

[1605] Device:

[1606] The terminal displays the received visualization data using a dedicated application, allowing the user to check graphs and charts.

[1607] Input: The submitted visualization data.

[1608] Output: The displayed graph or chart.

[1609] How it works: A dedicated application analyzes the data and provides visualized information to the user, allowing the user to quickly check the information they need through the application.

[1610] These steps enable users to quickly and accurately digitize paper-based documents and analyze and visualize data in real time.

[1611] (Application example 1)

[1612] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1613] Currently, small and medium-sized enterprises and sole proprietors face many challenges in quickly converting paper-based sales slips and expense receipts into digital data and using that data to make business decisions. Specifically, manual data entry requires time and effort and is prone to human error. Furthermore, real-time data analysis is difficult, making it difficult to immediately identify sales trends and wasteful expenses. This delays the optimization of inventory management and purchasing plans, as well as the development of cost-cutting measures. Therefore, there is a need for a system that can easily digitize this paper-based data and analyze and visualize it in real time.

[1614] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1615] In this invention, the server includes means for sending visualized results to the user's terminal and displaying them, means for processing the stored data based on an analytical algorithm, and means for storing and linking the received data in a database. This makes it possible to convert photos of sales slips and expense receipts taken by the user into digital data and visualize the analysis results, including sales trends and popular products, in real time.

[1616] "Paper-based documents" are records of information written on paper, such as sales slips and expense receipts.

[1617] The "image acquisition means" is a means for taking a photograph of a paper-based document using a camera such as a smartphone and acquiring the image.

[1618] An "optical character recognition engine" is software that extracts text from captured images and converts it into digital data.

[1619] The "means for formatting" is a means for classifying the converted text data into date, amount, category, etc., and formatting it into a predetermined format.

[1620] "Communication means" refers to a means for transmitting formatted data to a server via the Internet.

[1621] A "database" is a collection of digital data stored on a server, where each piece of data is appropriately tagged and linked.

[1622] An "analysis algorithm" is a calculation procedure used to analyze sales trends and popular products based on stored data.

[1623] "Means for visualizing analytical results" refers to means for visually representing the results obtained by analytical algorithms as graphs or charts.

[1624] The "user terminal" is a device such as a smartphone or tablet, which is used to display visualized analysis results.

[1625] "Sales trend" is an indicator that shows fluctuations in sales data over a certain period of time.

[1626] "Popular products" is an indicator of products with high sales during a specific period.

[1627] "Real-time" refers to the user instantly capturing and analyzing ongoing data.

[1628] This invention is a system that enables small and medium-sized enterprises and sole proprietors to easily digitize paper-based sales slips and expense receipts, and then analyze and visualize the data in real time. This system is composed of an image acquisition means, an optical character recognition engine, a data formatting means, a communication means, an analysis algorithm, and a means for visualizing the analysis results.

[1629] Hardware and software used

[1630] 1. Smartphone (device)

[1631] Image acquisition method:

[1632] Users use their smartphone to take a photo of a paper sales slip or expense receipt, and the image is captured directly within the application.

[1633] Optical Character Recognition Engine (OCR):

[1634] On the smartphone, the acquired image is converted into text data using an OCR engine (e.g., Tesseract).

[1635] 2. Server

[1636] Data formatting methods:

[1637] The converted text data is formatted by items such as date, amount, category, etc. before being sent to the server. The formatting process is carried out using a Python script on the smartphone.

[1638] Communication Method:

[1639] The formatted data is sent securely to the server using the HTTPS protocol.

[1640] Storage and linking methods:

[1641] The data received by the server is stored in a database and linked to the user's identification information. This database is typically MySQL.

[1642] Analysis algorithm:

[1643] Algorithms are applied to the data stored in the database to analyze sales trends and popular products, using Python or R.

[1644] Analysis results visualization methods:

[1645] The analyzed results are visualized using graphing libraries such as Matplotlib and D3.js.

[1646] 3. User's Device

[1647] Viewing Results:

[1648] The resulting data is analyzed and visualized on the server and sent to the user's smartphone in JSON format, where it is displayed to the user through a dedicated application.

[1649] Specific examples

[1650] 1. For retailers:

[1651] Retail store owners take photos of their daily sales slips with their smartphone camera. The images are converted into text data using an OCR engine, and then formatted into fields such as date, amount, and category. This data is securely sent to a server for storage and analysis. The analysis results are visualized as sales trends and can be viewed in real time on the user's smartphone.

[1652] 2. For sole proprietors:

[1653] Self-employed individuals take photos of expense receipts with their smartphone camera, and the device converts the images into text data using an OCR engine. This data is then formatted and sent to a server. The analysis results on the server are visualized as expense distribution by category and displayed on the user's smartphone. Self-employed individuals can use this data to plan cost-cutting measures.

[1654] Prompt Sentence Examples

[1655] You can enter a generative AI model using a prompt like this:

[1656] Please generate a concrete example application based on the following invention. The invention is a system that allows small and medium-sized enterprises and sole proprietors to easily digitize paper-based data and analyze and visualize the data in real time. This application is installed on a smartphone and takes photos of sales slips, expense receipts, etc., digitizes them, and then analyzes and visualizes the data based on that.

[1657] Although the embodiments for carrying out the present invention have been described in detail above, the scope of the present invention is not limited to the above examples.

[1658] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1659] Step 1:

[1660] A user takes a photo of a paper-based sales slip or expense receipt with their smartphone camera. This action generates image data. The input is a paper-based document, and the output is image data. Specifically, the user launches the smartphone's camera app and takes a photo of the document.

[1661] Step 2:

[1662] Once the captured image data is saved on the device, the device uses its internal optical character recognition engine (OCR) to convert this image data into text data. The input is image data, and the output is text data. Specifically, the OCR engine (e.g., Tesseract) recognizes the characters in the image and extracts them as text.

[1663] Step 3:

[1664] The converted text data is formatted by a program on the terminal based on the date, amount, category, etc. The input is the converted text data, and the output is the formatted data. Specifically, the formatting algorithm analyzes the text data and separates it into fields.

[1665] Step 4:

[1666] The formatted data is sent to the server in encrypted form using the HTTPS protocol. The input is the formatted data, and the output is the encrypted data transmission completion. Specifically, the terminal opens a secure communication channel and sends the data to the endpoint.

[1667] Step 5:

[1668] The data received by the server is first decrypted, linked to the user's account information, and then stored in a database. The input is encrypted data, and the output is digital data stored in the database. Specifically, the server decodes the data, links it to the corresponding user information, and stores it in the database.

[1669] Step 6:

[1670] The stored data is processed based on analytical algorithms on the server. For example, sales trends and popular products are analyzed. The input is digital data stored in a database, and the output is the analysis results. Specifically, Python or R scripts use SQL queries to retrieve the data and perform the analysis.

[1671] Step 7:

[1672] The analysis results are generated as graphs and charts using a visualization library (e.g., Matplotlib or D3.js). The input is the analysis results, and the output is the visualized data. Specifically, various diagrams and charts are generated based on the analysis results.

[1673] Step 8:

[1674] The visualized result data is sent from the server to the user's device in JSON format. The input is the visualized data, and the output is the completion of sending the JSON format data. Specifically, the server sends the data to the user's device via push notification or email.

[1675] Step 9:

[1676] The data received by the user's device is displayed by a dedicated application. The input is JSON formatted data, and the output is visualized data displayed on the user interface. Specifically, the application parses the JSON data and displays it in an appropriate format.

[1677] Step 10:

[1678] Based on this visualized data, users make business decisions such as inventory management, adjusting purchasing plans, and planning cost reduction measures. The input is the visualized data, and the output is the user's decision. Specifically, users check the data on the application and take action as needed.

[1679] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1680] The present invention combines a system that allows small and medium-sized enterprises and sole proprietors to easily digitize paper-based data, analyze and visualize the data in real time, and further combines it with an emotion engine that recognizes the user's emotions. This system is implemented as follows.

[1681] 1. Data capture (camera capture of paper-based documents)

[1682] User:

[1683] The user launches the camera app on their smartphone, takes a photo of a paper-based document (e.g., a sales ledger or expense receipt), checks the appropriate position and lighting, and presses the shutter button.

[1684] Device:

[1685] The captured image is converted into digital text by the device's optical character recognition (OCR) engine, which then formats the text into a specified format. This formatting process includes sorting the data by date, amount, category, and other items.

[1686] 2. Data transmission and storage

[1687] Device:

[1688] The formatted data is encrypted and sent securely from the device to the server using the HTTPS protocol.

[1689] server:

[1690] Once the data arrives at the server, it is first decrypted and then stored in a database based on the user's account information. During the storage process, the data is verified for integrity and placement in the correct location.

[1691] 3. Data analysis and visualization

[1692] server:

[1693] Data stored in a database is retrieved periodically or upon user request and predefined analytical algorithms are applied, for example, to analyze sales trends or expense distribution.

[1694] server:

[1695] The analysis results are generated in a visually friendly format, using graphing libraries such as Matplotlib and D3.js to create bar charts, pie charts, line graphs, etc.

[1696] 4. Sending and displaying results

[1697] server:

[1698] The generated visualization data is sent to the user's device in JSON format, and the user is notified via push notification or email.

[1699] Device:

[1700] The received visualization data is displayed in a dedicated application, and users can view this data through the application interface and quickly check the information they need.

[1701] 5. Use of Emotion Engine

[1702] Device:

[1703] The emotion engine recognizes the user's facial expressions and voice, and acquires the user's emotion data in real time. The emotion engine uses machine learning algorithms to identify emotions (e.g., joy, anger, sadness, surprise).

[1704] server:

[1705] The acquired emotional data is incorporated into the analysis results, and the information provided takes into account the user's emotional state. For example, if the user expresses dissatisfaction, the system will provide a detailed explanation or additional advice.

[1706] Specific examples

[1707] For retailers

[1708] The user, a retail store owner, takes a photo of their daily sales ledger with their smartphone camera. The device sends the image to an OCR engine, which converts and formats it into sales data. This data is securely sent to a server, where it is stored and analyzed in a database. The analysis results are visualized as a sales trend graph and displayed on the user's smartphone. The emotion engine analyzes the owner's facial expressions, and if a positive emotion is recognized, the system shares success stories and provides further advice.

[1709] For sole proprietors

[1710] Self-employed individuals take photos of expense receipts with their smartphone camera, and the device converts and formats the images into text data using an OCR engine. The converted and formatted expense data is sent to a server and stored in a database. The server analyzes the distribution of expenses by category and sends the results, visualized as a pie chart, to the user's device. If the emotion engine recognizes the user's stress or anxiety, content offering specific ways to reduce costs and mitigation measures is displayed.

[1711] This allows users to receive not just data analysis but also emotionally-based advice, helping them make more effective business decisions.

[1712] The processing flow will be explained below.

[1713] Step 1:

[1714] The user launches the camera app on their smartphone and takes a photo of a paper-based document (e.g., a sales ledger or expense receipt). The user checks the appropriate position and lighting, then presses the shutter button.

[1715] Step 2:

[1716] The device sends the captured image to an optical character recognition (OCR) engine, which extracts the characters in the image as digital text. The OCR engine analyzes the characters in the image and converts the data into text.

[1717] Step 3:

[1718] The device then formats the text data converted by OCR into a specific format, which involves sorting the data by date, amount, category, etc., and converting it into the appropriate format.

[1719] Step 4:

[1720] The device encrypts the formatted data and sends it securely to the server, using the HTTPS protocol to ensure secure data transmission.

[1721] Step 5:

[1722] The server receives the data, decrypts it, associates it with a specific user account, and stores it in a database, where the data is verified for integrity and placement in the correct location.

[1723] Step 6:

[1724] The device transmits the user's facial expressions and voice to the emotion engine, which uses machine learning algorithms to identify the user's emotions (e.g., joy, anger, sadness, surprise) in real time.

[1725] Step 7:

[1726] The server retrieves business data stored in the database periodically or upon user request and applies predefined analytical algorithms to it, for example, analyzing sales trends or expense distribution.

[1727] Step 8:

[1728] The server integrates the analysis results with the user's emotional data and generates the analysis results in a visually easy-to-understand format, such as a bar graph, pie chart, or line graph.

[1729] Step 9:

[1730] The server sends the generated visualization data to the user's device in JSON format, and the user is notified via push notification or email.

[1731] Step 10:

[1732] The device analyzes the received visualization data and displays it in a dedicated application. Users can view graphs and charts through the application interface and quickly obtain the information they need.

[1733] Step 11:

[1734] The server dynamically adjusts the display content based on the user's emotional data. For example, if the user is feeling anxious, the analysis results will include detailed explanations and advice.

[1735] Step 12:

[1736] Users can make business decisions based on the displayed data and additional advice. For example, they can manage inventory based on sales trends or consider cost-cutting measures based on expense distribution. The system also includes advice based on emotional data, allowing users to make effective decisions while reducing psychological burden.

[1737] Example 2

[1738] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1739] For small and medium-sized enterprises and sole proprietors, digitizing paper-based data and analyzing and visualizing it in real time is a time-consuming and labor-intensive task. Furthermore, conventional systems lack the functionality to provide personalized advice that takes into account the user's emotions. As a result, users are limited to simply visualizing the data, making it difficult for them to make effective, emotion-based decisions.

[1740] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an image acquisition means for capturing a paper-based document, a means for converting the captured image into text data using an optical character recognition engine, a means for formatting the converted text data into a specific format, a communication means for transmitting the formatted data to a remote server, a means for storing the received data in a database and associating it with user information, a means for processing the stored data based on an analysis algorithm, a means for visually displaying the analysis results, a means for transmitting the visualized results to the user's device and displaying them, a means for using an emotion recognition engine for analyzing the user's emotions, and a means for including emotion data in the analysis results and providing information tailored to the user. This allows the user to easily digitize paper-based data, analyze and visualize the data in real time, and receive personalized advice based on their emotions.

[1741] "Paper-based documents" refers to text or data that is written on physical paper.

[1742] "Image capture means" refers to a device or method for capturing a paper-based document as a digital image.

[1743] "Optical character recognition engine" refers to software or algorithms that analyze character information from image data and convert it into digital text.

[1744] "Formatting means" refers to processing means that rearranges acquired digital text into a particular format or structure to facilitate storage in a database.

[1745] "Communication means" refers to the network protocols and devices used to transmit data to a remote server.

[1746] "Server" refers to a computer system for storing, processing, and managing data.

[1747] "Database" refers to a data storage system for efficiently storing, retrieving, and managing structured data.

[1748] An "analytical algorithm" refers to a set of computational procedures or rules for processing stored data and extracting useful information.

[1749] "Visualization means" refers to tools and methods for displaying extracted information in an easy-to-understand format, such as graphs or charts.

[1750] "User device" refers to a device (e.g., a smartphone or tablet) that a user owns and uses to display and interact with information.

[1751] An "emotion recognition engine" refers to software or algorithms that analyze and identify emotions from a user's facial expressions and voice.

[1752] "Emotion data" refers to data that indicates the emotional state of the user analyzed by an emotion recognition engine.

[1753] "Information provision" refers to the act of presenting necessary information or advice to a user.

[1754] This invention combines a system that allows small and medium-sized enterprises and individual business owners to easily digitize paper-based data, analyze and visualize the data in real time, and an emotion engine that recognizes user emotions. This system is realized by a specific combination of hardware and software.

[1755] Data capture (camera capture of paper-based documents)

[1756] User:

[1757] A user starts a camera app on a smartphone and takes a picture of a paper-based document (e.g., a sales ledger or expense receipt). For example, the user takes a picture of the sales ledger to obtain a digital image.

[1758] Device:

[1759] The device receives the captured image data and converts it into digital text using an optical character recognition engine (e.g., Google Cloud Vision or Tesseract OCR), then uses the Python Pandas library to format the text data into a specific format (date, amount, category, etc.).

[1760] Data transmission and storage

[1761] Device:

[1762] The formatted data is encrypted and sent to the server using the HTTPS protocol. The SSL / TLS protocol is used for encryption. For example, dates, sales data, etc. are sent securely.

[1763] server:

[1764] The received data is decrypted and stored in a database (e.g., PostgreSQL) based on the user's account information using a web framework such as Django or Flask. The data is verified for consistency and placement in the appropriate fields.

[1765] Data Analysis and Visualization

[1766] server:

[1767] The stored data is retrieved periodically or upon user request and predefined analytical algorithms are applied using machine learning libraries such as Scikit-learn and TensorFlow, for example to analyze sales trends or expense distribution.

[1768] server:

[1769] The analysis results are generated in visually easy-to-understand formats (bar graphs, pie charts, line graphs) using graphing libraries such as Matplotlib and D3.js.

[1770] Sending and displaying results

[1771] server:

[1772] The generated visualization data is sent to the user's device in JSON format, and a message such as "Monthly sales report has been generated" is provided via push notification or email.

[1773] Device:

[1774] The received data is displayed in a dedicated application using frameworks such as React Native or Flutter, and users can view graphs and charts through the app interface.

[1775] Using the Emotion Engine

[1776] Device:

[1777] The system captures the user's facial expressions and voice in real time using Microsoft Azure's Face API and Google Cloud Speech-to-Text API, allowing it to analyze the user's emotions (e.g., joy, anger, sadness, surprise).

[1778] server:

[1779] The acquired emotional data is then incorporated into the analysis results to provide personalized advice and information based on the user's emotional state. For example, the system may provide a message such as, "You seem pleased with your recent sales growth. Would you like to share your success story?"

[1780] Specific examples

[1781] For retailers

[1782] User:

[1783] A retail store owner takes a photo of their daily sales ledger using their smartphone camera.

[1784] Device:

[1785] The device sends the image to the OCR engine, which converts and formats it into sales data.

[1786] server:

[1787] The data stored in the database is analyzed and visualized as a sales trend graph. The visualization results are sent to the user's smartphone and displayed in a dedicated application.

[1788] Emotion Engine:

[1789] The emotion engine analyzes the user's facial expressions, and if a positive emotion is recognized, it will share success stories and provide further advice.

[1790] For sole proprietors

[1791] User:

[1792] A self-employed person takes a photo of an expense receipt with their smartphone camera.

[1793] Device:

[1794] The device converts and formats the image into text data using an OCR engine.

[1795] server:

[1796] The data stored in the database is analyzed, and the distribution of expenses by category is visualized as a pie chart and sent to the user's device.

[1797] Emotion Engine:

[1798] The emotion engine recognizes the user's stress and anxiety and displays content offering specific ways to reduce costs and provide relief.

[1799] Prompt Sentence Examples

[1800] By inputting the following prompt sentence into the generative AI model, an explanation of the system design and processing described above can be generated.

[1801] "Design a system that allows small and medium-sized businesses and sole proprietors to easily digitize paper-based data and analyze and visualize the data in real time. This system should also incorporate an emotion engine that recognizes the user's emotions. Explain the system's processing in natural language by following the steps and providing concrete examples."

[1802] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1803] Step 1: Data capture (camera capture of paper-based documents)

[1804] User:

[1805] A user launches the camera app on their smartphone and takes a picture of a paper-based document (e.g., a sales ledger or expense receipt), checks whether the image is clear, and presses the shutter button. The input is the paper-based document, and the output is a digital image.

[1806] Device:

[1807] The device saves the captured image data and launches an OCR engine (such as Google Cloud Vision or Tesseract OCR). The input is a digital image, and the output is text data, which includes the date and amount.

[1808] Step 2: Formatting the text data

[1809] Device:

[1810] The text data obtained by the OCR engine is formatted using Python's Pandas library. The input is raw text data, and the output is formatted data, for example, "Date: 2023 / 10 / 01, Sales: 20,000 yen." Here, the text data is classified into specific items (date, amount, category, etc.).

[1811] Step 3: Sending data

[1812] Device:

[1813] The formatted data is encrypted using AES and sent to the server using the HTTPS protocol. The input is the formatted data and the output is the encrypted data. The SSL / TLS protocol is used for transmission.

[1814] Step 4: Decrypt and save the data

[1815] server:

[1816] Receives encrypted data and first decrypts it. The input is the encrypted data and the output is the decrypted data. Next, using a web framework such as Django or Flask, the data is saved in a database (e.g., PostgreSQL) based on the user's account information. The input is the decrypted data and the output is the data saved in the database.

[1817] Step 5: Acquire and analyze data

[1818] server:

[1819] The stored data is retrieved periodically or upon user request. The input is the data stored in the database, and the output is the retrieved data. Then, using a machine learning library such as Scikit-learn or TensorFlow, a predefined analytical algorithm is applied. For example, to analyze sales trends or expense distribution. The input is the retrieved data, and the output is the analysis results.

[1820] Step 6: Visualize the data

[1821] server:

[1822] The analysis results are generated in a visually easy-to-understand format (bar graph, pie chart, line graph) using graph creation libraries such as Matplotlib and D3.js. The input is the analysis results, and the output is the visualized data.

[1823] Step 7: Send and display visualization data

[1824] server:

[1825] The generated visualization data is sent to the user's device in JSON format. The input is the visualization data, and the output is the transmitted data. A message such as "Monthly sales report has been generated" is provided via push notification or email.

[1826] Device:

[1827] The received data is displayed in a dedicated application using a framework such as React Native or Flutter. The input is the transmitted JSON data, and the output is a graph or chart.

[1828] Step 8: Emotion recognition and advice provision

[1829] Device:

[1830] The user's facial expressions and voice are acquired in real time using Microsoft Azure's Face API and Google Cloud Speech-to-Text API. The input is the user's facial expressions and voice, and the output is emotion data. The emotion data is sent to the server upon request.

[1831] server:

[1832] Receives emotion data and incorporates it into the analysis results. For example, if the user is satisfied, relevant advice is provided. The input is emotion data, and the output is the advice the user applies. The advice is sent to the user's device in JSON format and displayed in a React Native or Flutter application.

[1833] (Application example 2)

[1834] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1835] Currently, many small and medium-sized enterprises and sole proprietors handle a large amount of paper-based documents, and digitizing this data is a time-consuming and labor-intensive process. While there is also a need for analysis and visualization to effectively utilize digital data and support decision-making, current systems make it difficult to do this quickly and accurately. Furthermore, in workplaces where responses must take into account the emotional state of employees, there is a lack of means to grasp employees' stress and anxiety, leading to reduced production efficiency and safety risks. A system that can solve these issues is needed.

[1836] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing a user's emotion in real time, means for incorporating the recognized emotion data into the analysis result, and means for providing information based on the user's emotional state. This makes it possible not only to quickly digitize paper-based documents and perform data analysis and visualization, but also to grasp the user's emotional state in real time using an emotion engine and provide appropriate information.

[1837] "Image capture means for capturing images of paper-based documents" refers to devices and techniques used to capture image data of paper documents.

[1838] An "optical character recognition engine" is software or hardware that analyzes character information in image data and converts it into digital text data.

[1839] "Means for formatting converted text data" refers to a process or device for formatting the text data obtained by the OCR engine into a specific format.

[1840] The "communication means for transmitting formatted data to a server" is a technology for securely transmitting formatted digital data to a server via a network.

[1841] "Means for storing and linking received data in a database" refers to a technology that allows the server to store data received in a database and link it to related information.

[1842] The "means for processing stored data based on an analytical algorithm" is a technology for processing information stored in a database based on a specific analytical algorithm.

[1843] "Means for visualizing analysis results" refers to techniques for visually displaying the data obtained through analysis in the form of graphs or charts in an easy-to-understand manner.

[1844] The "means for transmitting the visualized results to the user's terminal and displaying them" refers to a technique for transmitting the visualized data to the user's device and displaying it on that device.

[1845] The "emotion recognition means for recognizing the user's emotions" is a technology used to analyze the user's facial expressions and voice to determine the user's emotional state.

[1846] "Means for incorporating emotional data into analytical results" refers to a process or technique for adding acquired emotional data to analytical results.

[1847] "Means for providing information based on emotional state" refers to a technique for providing appropriate information or advice according to the user's emotional state.

[1848] This invention combines a system that allows small and medium-sized enterprises and sole proprietors to easily digitize paper-based data, analyze and visualize the data in real time, with an emotion engine that recognizes the user's emotions. To specifically implement this system, the following elements are required:

[1849] 1. Image acquisition method

[1850] The user activates the camera on their smart glasses or smartphone and takes a picture of a paper-based document, which captures the data on the paper medium as image data.

[1851] 2. Optical character recognition means

[1852] The captured image is then analyzed through an optical character recognition engine such as Tesseract OCR, which converts the textual information in the image into digital text data. This process also requires pre-processing techniques to improve the accuracy of image identification.

[1853] 3. Data Formatting Methods

[1854] The converted text data is formatted by date, amount, category, etc. This process is automated by a program to ensure data consistency.

[1855] 4. Means of communication

[1856] The formatted data is securely transmitted to the server using the HTTPS protocol, where it is encrypted to ensure the protection of the information.

[1857] 5. Data storage means

[1858] The server stores the received data in a database and associates it with the user's account information, thereby maintaining data integrity.

[1859] 6. Data Analysis Methods

[1860] The stored data is processed through Python analytical algorithms to analyze sales trends, expense distribution, etc. The analysis results are visualized using Matplotlib and D3.js.

[1861] 7. Results display means

[1862] The visualized results are sent to the user's device in JSON format and are notified to the user via push notification or email. The user can view this data through a dedicated application.

[1863] 8. Emotion recognition means

[1864] Using emotion recognition engines such as "Affectiva," the system extracts emotional data from the user's facial expressions and voice. The emotional data is then incorporated into the analysis results, and information and advice is provided based on the user's emotional state.

[1865] Specific examples

[1866] Factory workers take a photo of their daily reports using smart glasses. OCR converts the reports into digital data and sends it to a server. The data is then analyzed and visualized as an efficiency graph created in Python. Emotion recognition technology also monitors employee stress and sends alerts to managers if necessary.

[1867] Prompt Sentence Examples

[1868] "Smart glasses used on factory floors can take photos of paper-based reports, convert them into text using OCR, and then format the data and send it to a server. Then, data analysis and visualization can be performed, and an emotion engine can be used to detect and respond to employee stress."

[1869] This system not only streamlines work processes in factories, but also allows for flexible responses that take into account the emotional state of employees.

[1870] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1871] Step 1:

[1872] A user takes a photo of a paper-based document using the camera on their smart glasses or smartphone. This image capture means captures the paper-based document as digital image data. The input is the paper-based document, and the output is digital image data. Specifically, the user confirms the appropriate position and lighting, and then presses the shutter button to capture the image.

[1873] Step 2:

[1874] Using the image captured by the device, the optical character recognition (OCR) engine "Tesseract OCR" analyzes the characters in the image and converts them into text data. The input is digital image data, and the output is text data. Specifically, the OCR engine analyzes the image, extracts the character information from the image, and converts it into digital text.

[1875] Step 3:

[1876] The terminal formats the converted text data into items such as date, amount, category, etc. The input is the converted text data, and the output is the formatted data. Specifically, the program automatically analyzes the text data and formats it into the specified format.

[1877] Step 4:

[1878] The terminal sends the formatted data to the server using the HTTPS protocol. This communication method ensures that the formatted data is securely sent to the server. The input is the formatted data, and the output is the completion of data transmission to the server. Specifically, the data is encrypted and sent via HTTPS.

[1879] Step 5:

[1880] The server stores the received data in a database and associates it with the user's account information. This data storage means ensures that the data is consistent and placed in the appropriate location. The input is the data sent to the server, and the output is stored in the database. Specifically, the server analyzes the data and associates it with the user's account information before storing it.

[1881] Step 6:

[1882] The server processes the data stored in the database by applying analytical algorithms. This data analysis method analyzes sales trends, expense distribution, and so on. The input is the data stored in the database, and the output is the analysis results. Specifically, a data analysis algorithm using Python processes the data and generates the results.

[1883] Step 7:

[1884] The server visualizes the analysis results and displays them in the form of graphs and charts using "Matplotlib" and "D3.js." This visualization method makes the analysis results visually easy to understand. The input is the analysis results, and the output is visualized data. Specifically, the server retrieves the analysis results and generates graphs and charts for visual display.

[1885] Step 8:

[1886] The server sends the visualized results to the user's device and displays them in a dedicated application. This means of displaying the results allows the user to easily check the analysis results. The input is the visualized data, and the output is the completion of transmission to the user's device. Specifically, the server generates data in JSON format and sends it to the user's device.

[1887] Step 9:

[1888] The device uses an emotion recognition engine such as "Affectiva" to acquire emotional data from the user's facial expressions and voice. This emotion recognition means determines the user's emotional state in real time. The input is the user's facial expressions and voice, and the output is emotional data. Specifically, the device acquires the user's facial expressions and voice through a camera or microphone and analyzes them using an emotion recognition algorithm.

[1889] Step 10:

[1890] The server incorporates the acquired emotional data into the analysis results and provides information based on the user's emotional state. This emotional data integration means enables responses that correspond to the user's emotions. The inputs are emotional data and analysis results, and the output is integrated information. Specifically, the server uses the emotional data to reevaluate the analysis results and provide additional advice or information.

[1891] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1892] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1893] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1894] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1895] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1896] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1897] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1898] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1899] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1900] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1901] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1902] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1903] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1904] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1905] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1906] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1907] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1908] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1909] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1910] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1911] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1912] The following is further disclosed regarding the above embodiment.

[1913] (Claim 1)

[1914] [an image capture means for capturing an image of a paper-based document;

[1915] [Means for converting the acquired image into text data using an optical character recognition engine;

[1916] [Means for formatting the converted text data;

[1917] [a communication means for transmitting the formatted data to a server;

[1918] [Means for storing and linking received data in a database;

[1919] [Means for processing the stored data based on analytical algorithms; and

[1920] [Means for visualizing the analysis results and

[1921] [Means for transmitting and displaying the visualized results on a user's terminal; and

[1922] A system including:

[1923] (Claim 2)

[1924] [processing means for converting and shaping the captured image;

[1925] [a processing means for securely transmitting the converted and formatted data to a server;

[1926] [a processing means for retrieving and analyzing data from the database;

[1927] [processing means for visualizing and transmitting the analysis results;

[1928] 10. The system of claim 1, comprising:

[1929] (Claim 3)

[1930] [a terminal means including a means for formatting text data acquired by an optical character recognition engine;

[1931] [a terminal means having a communication means for transmitting the formatted data to a server;

[1932] [a server means having means for storing and analyzing received data;

[1933] [a server means having means for displaying the analysis results on a user's terminal;

[1934] 10. The system of claim 1, comprising:

[1935] "Example 1"

[1936] (Claim 1)

[1937] [an image capture means for capturing an image of a paper-based document;

[1938] [Means for converting the acquired image into text data using an optical character recognition engine;

[1939] [Means for formatting the converted text data into a specified format;

[1940] [a communication means for encrypting the formatted data and transmitting it to a server;

[1941] [Means for decoding data received using a communication protocol and storing it in a database;

[1942] [Means for applying and processing analytical algorithms based on the stored data; and

[1943] [Means for visualizing the analysis results as graphs and charts;

[1944] [Means for transmitting the visualized result data to a user's terminal and displaying it;

[1945] A system including:

[1946] (Claim 2)

[1947] [a processing means for converting and shaping the captured image;

[1948] [a processing means for encrypting the converted and formatted data and transmitting it securely to a server;

[1949] [Means for retrieving data from a database and processing it by applying analytical algorithms;

[1950] [a processing means for visualizing and transmitting the analysis results;

[1951] 10. The system of claim 1, comprising:

[1952] (Claim 3)

[1953] [A terminal means including a means for formatting text data acquired by an optical character recognition engine into a specified format;

[1954] [A terminal means having a communication means for encrypting the formatted data and transmitting it to a server;

[1955] [a server means having means for decrypting the received data, storing it in a database, and analyzing it;

[1956] [a server means having means for visualizing the analysis results as graphs or charts and transmitting them to a user's terminal;

[1957] 10. The system of claim 1, comprising:

[1958] "Application Example 1"

[1959] (Claim 1)

[1960] [an image capture means for capturing an image of a paper-based document;

[1961] [Means for converting the acquired image into text data using an optical character recognition engine;

[1962] [Means for formatting the converted text data;

[1963] [a communication means for transmitting the formatted data to a server;

[1964] [Means for storing and linking received data in a database;

[1965] [Means for processing the stored data based on analytical algorithms; and

[1966] [Means for visualizing the analysis results and

[1967] [Means for transmitting and displaying the visualized results on a user's terminal; and

[1968] [A means to convert sales slips and expense receipts photographed by users into digital data and visualize analytical results, including sales trends and popular products, in real time.

[1969] A system including:

[1970] (Claim 2)

[1971] [processing means for converting and shaping the captured image;

[1972] [a processing means for securely transmitting the converted and formatted data to a server;

[1973] [a processing means for retrieving and analyzing data from the database;

[1974] [processing means for visualizing and transmitting the analysis results;

[1975] [A processing means for providing information for adjusting inventory management and purchasing plans based on the visualized data;

[1976] 10. The system of claim 1, comprising:

[1977] (Claim 3)

[1978] [a terminal means including a means for formatting text data acquired by an optical character recognition engine;

[1979] [a terminal means having a communication means for transmitting the formatted data to a server;

[1980] [a server means having means for storing and analyzing received data;

[1981] [a server means having means for displaying the analysis results on a user's terminal;

[1982] [A terminal means including a means for visualizing the analysis results as graphs and charts including sales trends and popular products to support user decision-making;

[1983] 10. The system of claim 1, comprising:

[1984] "Example 2: Combining Emotion Engines"

[1985] (Claim 1)

[1986] [an image acquisition means for capturing an image of a paper-based document;

[1987] [Means for converting the acquired image into text data using an optical character recognition engine;

[1988] [Means for formatting the converted text data into a specific format;

[1989] [a communication means for transmitting the formatted data to a remote server;

[1990] [Means for storing the received data in a database and associating it with user information;

[1991] [Means for processing the stored data based on analytical algorithms; and

[1992] [Means for visually displaying the analysis results;

[1993] [means for transmitting and displaying the visualized results on a user's device;

[1994] [Means for utilizing an emotion recognition engine to analyze the user's emotions;

[1995] [Means for including emotion data in the analysis results and providing information adapted to the user; and

[1996] A system including:

[1997] (Claim 2)

[1998] [Means for executing a process for converting and shaping the captured image] The system according to claim 1.

[1999] (Claim 3)

[2000] [Means for executing a process for securely transmitting the converted and formatted data to a server] The system according to claim 1.

[2001] (Claim 4)

[2002] [Means for executing a process of retrieving and analyzing data from a database] The system according to claim 1.

[2003] (Claim 5)

[2004] [Means for executing the means for acquiring and analyzing emotion data by an emotion recognition engine] The system according to claim 1.

[2005] (Claim 6)

[2006] The system according to claim 1, further comprising: a means for executing a means for using the analyzed emotion data to provide information adapted to the user.

[2007] "Application example 2 when combining emotion engines"

[2008] (Claim 1)

[2009] [an image capture means for capturing an image of a paper-based document;

[2010] [Means for converting the acquired image into text data using an optical character recognition engine;

[2011] [Means for formatting the converted text data;

[2012] [a communication means for transmitting the formatted data to a server;

[2013] [Means for storing and linking received data in a database;

[2014] [Means for processing the stored data based on analytical algorithms; and

[2015] [Means for visualizing the analysis results and

[2016] [Means for transmitting and displaying the visualized results on a user's terminal; and

[2017] [an emotion recognition means for recognizing an emotion of a user;

[2018] [Means for incorporating emotion data into analysis results;

[2019] [Means for providing information based on emotional states;

[2020] A system including:

[2021] (Claim 2)

[2022] [processing means for converting and shaping the captured image;

[2023] [a processing means for securely transmitting the converted and formatted data to a server;

[2024] [a processing means for retrieving and analyzing data from the database;

[2025] [processing means for visualizing and transmitting the analysis results;

[2026] [A processing means for recognizing user emotions and integrating them with analysis results to provide information;

[2027] 10. The system of claim 1, comprising:

[2028] (Claim 3)

[2029] [a terminal means including a means for formatting text data acquired by an optical character recognition engine;

[2030] [a terminal means having a communication means for transmitting the formatted data to a server;

[2031] [a server means having means for storing and analyzing received data;

[2032] [a server means having means for displaying the analysis results on a user's terminal;

[2033] [A terminal and a server means equipped with emotion recognition means for recognizing a user's emotion and providing appropriate information;

[2034] 10. The system of claim 1, comprising: [Explanation of symbols]

[2035] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. image capture means for capturing an image of a paper-based document; A means for converting the acquired image into text data using an optical character recognition engine; means for formatting the converted text data; A communication means for transmitting the formatted data to a server; A means for storing and linking the received data in a database; means for processing the stored data based on an analytical algorithm; a means of visualizing the results of the analysis; means for transmitting the visualized results to a user's terminal for display; A system including:

2. processing means for transforming and shaping the captured image; processing means for securely transmitting the converted and formatted data to a server; processing means for retrieving and analyzing data from the database; processing means for visualizing and transmitting the analysis results; 10. The system of claim 1, comprising:

3. a terminal means including means for formatting the text data acquired by the optical character recognition engine; a terminal means having a communication means for transmitting the formatted data to a server; server means having means for storing and analyzing received data; a server means having means for displaying the analysis results on a user's terminal; 10. The system of claim 1, comprising:

Images