System

Abstract

A system is provided.SOLUTION: A system comprising: means for a user to capture an image of a paper document and upload the image to the system; means for a server to receive the image and extract text data in the image using optical character recognition means; means for the server to analyze the extracted text data and layout of the image and automatically generate a digital input form; and means for the server to transmit the generated digital input form to the user's terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Working with paper documents requires a lot of time and effort, and this burden is particularly heavy in workplaces with labor shortages. Furthermore, manually inputting data from paper documents into a PC is prone to input errors, and correcting these errors further increases the amount of work required. Companies and public institutions, in particular, handle vast amounts of paper documents every day, and input errors and inefficient work can cause significant delays in operations. There is a need for a system that can solve these problems, improve work efficiency, and achieve accurate data management. [Means for solving the problem]

[0005] The present invention provides a system including a means for a user to take an image of a paper document and upload the image to a system, a means for a server to receive the image and extract text data from the image using optical character recognition, a means for the server to analyze the extracted text data and the layout of the image to automatically generate a digital input form, a means for transmitting the generated digital input form to the user's terminal and displaying it on the terminal, and a means for transmitting data entered by the user on the terminal to the server, where the server stores the data in a database. The system also provides a system that further includes a means for analyzing the position and structure of columns in order to reproduce the layout of the original paper document when generating the digital input form, and a means for protecting the image data using encryption when the terminal transmits the captured image to the server. This allows for rapid and accurate digitization of paper documents, improving business efficiency and ensuring accurate data management.

[0006] A "user" is an entity that takes an image of a paper document and uploads it to the system.

[0007] A "terminal" is a device used by a user that has the function of taking and saving images of paper documents and transmitting them to a server.

[0008] A "server" is a computer system that has the functionality to receive images sent by a user, perform optical character recognition and layout analysis, automatically generate a digital input form, and send it to the user's terminal.

[0009] "Images" are photographs of paper documents that are stored in digital format and uploaded to the system.

[0010] "Optical character recognition" is a technique for analyzing text within an image and extracting character data in digital form.

[0011] "Text data" is textual information extracted from an image using optical character recognition.

[0012] "Layout analysis" is the process of analyzing the position and structure of text and fields within an image of a paper document to obtain information for generating a digital input form.

[0013] A "digital input form" is an input form that has the same layout as a paper document, and allows users to input data on the screen.

[0014] "Encryption" is a technology that converts the contents of data to protect it so that it cannot be deciphered by third parties.

[0015] A "database" is a data management system that systematically manages data stored on a server, making it easy to search and use. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention is a system that digitizes paper documents, eliminates the need for manual data entry, and enables efficient and accurate information collection. This system functions as follows.

[0038] Program processing overview

[0039] Image upload

[0040] 1. The user takes a picture on the device

[0041] Users use the camera on their smartphone or tablet to take a picture of a paper document, such as an invoice or receipt.

[0042] 2. User uploads image to the system

[0043] Users upload the images they have taken to the system through a dedicated application. To do so, they press the "upload" button to send the image files to the server.

[0044] Image processing and analysis

[0045] 3. The server receives the image

[0046] The server receives the image sent from the user's device and temporarily stores it for further processing.

[0047] 4. The server performs optical character recognition (OCR)

[0048] The server runs an OCR engine on the received image, analyzes the text information in the image, and extracts the text data in a digital format, possibly using an OCR library such as Tesseract.

[0049] 5. The server performs layout analysis

[0050] The server analyzes the layout based on the location information of the text data and identifies the structure of the input form, allowing the digital input form to be reproduced with the same layout as the paper document.

[0051] Input form generation

[0052] 6. The server generates the digital input form

[0053] The server generates a template based on the analysis results and creates an input form with a layout similar to the original document.

[0054] 7. The server sends the input form to the device

[0055] The server sends the generated digital input form to the user's device, where it is encoded in JSON or HTML and displayed on the user's device.

[0056] Data entry and saving

[0057] 8. The user enters data on the device

[0058] The user enters data into a digital input form displayed on the terminal, such as invoice item names, quantities, and amounts.

[0059] 9. User submits input data

[0060] After completing the data entry, the user presses the "Submit" button to send the entered data to the server.

[0061] 10. The server stores the data

[0062] The server stores the received data in a database and manages the necessary information for subsequent processing and report generation.

[0063] Specific examples

[0064] Retailer Use

[0065] For example, this system is very useful for retailers to easily digitize their daily sales data. Retailers' staff take photos of each customer's receipt with their smartphones and upload them. The server analyzes the images and generates a digital receipt entry form. The staff enters the data on the device and saves it on the server, making it easy to digitize sales data.

[0066] Use of medical facilities

[0067] In medical facilities, patient medical records are often kept on paper. By using this system, medical records are digitized and managed efficiently. Medical staff take photos of paper medical records and upload them to the system. The server analyzes the images and generates a digital medical record entry form. Staff enter the necessary data into the form and save it on the server, furthering the digitization of medical records.

[0068] These processes enable paper documents to be digitized quickly and accurately, resulting in more efficient operations and accurate data management.

[0069] The processing flow will be explained below.

[0070] Step 1:

[0071] The user takes a photo on the device

[0072] Users use the camera on their smartphone or tablet to take a picture of a paper document (such as an invoice or receipt) using either a dedicated camera app or the device's standard camera app.

[0073] Step 2:

[0074] User uploads an image to the system

[0075] The user launches a dedicated application, selects the captured image, and presses the "upload" button to send the image to the system.

[0076] Step 3:

[0077] The device sends the image to the server

[0078] The device uses an HTTP request (such as a POST request) to send the captured image to the server, compressing or encrypting the image data as necessary.

[0079] Step 4:

[0080] The server receives the image

[0081] The server receives the image sent by the user, temporarily stores it in storage, and returns a response confirming receipt to the terminal.

[0082] Step 5:

[0083] The server runs the OCR engine

[0084] The server runs an optical character recognition (OCR) engine on the received image to extract text data from the image. For example, if the server uses the Tesseract OCR engine, it calls its API to process the data.

[0085] Step 6:

[0086] The server performs the layout analysis

[0087] The server analyzes the document layout based on the location information of the extracted text data, in order to identify the structure of tables and the positions of input fields, and obtain data for generating a digital input form.

[0088] Step 7:

[0089] The server generates the digital input form

[0090] The server generates a digital input form based on the analysis results, which has the same layout (position of input fields, item names, etc.) as the original paper document.

[0091] Step 8:

[0092] The server encodes the input form

[0093] The server encodes the generated digital input form in a format such as JSON or HTML and prepares it to be sent to the user's device.

[0094] Step 9:

[0095] The server sends the input form to the terminal

[0096] The server transmits the encoded digital input form data to the user's terminal, including the transmitted data as a response.

[0097] Step 10:

[0098] The device receives the input form

[0099] The terminal receives and decodes the digital input form data sent from the server, thereby preparing to display the digital input form on the terminal.

[0100] Step 11:

[0101] The device displays the input form

[0102] The terminal displays the received digital input form on the screen so that the user can fill it in. The user interface is designed to be intuitive.

[0103] Step 12:

[0104] The user enters data into the input form

[0105] The user inputs necessary data into the digital input form displayed on the terminal, such as product name, quantity, and price.

[0106] Step 13:

[0107] User saves or submits input data

[0108] After completing the data entry, the user presses the "Save" or "Send" button. If the "Save" button is pressed, the data is temporarily saved on the terminal, and if the "Send" button is pressed, the data is sent to the server.

[0109] Step 14:

[0110] The device sends the input data to the server

[0111] The terminal sends the input data to the server via an HTTP request, encrypting the data if necessary.

[0112] Step 15:

[0113] The server receives and stores the data

[0114] The server validates the data received from the devices and stores it in a database for further processing and report generation.

[0115] Step 16:

[0116] The server performs data analysis and report generation

[0117] The server performs statistical processing and generates reports based on the stored data, such as aggregating sales data and generating monthly reports.

[0118] Through these steps, paper documents can be digitized quickly and accurately, resulting in more efficient operations and accurate data management.

[0119] Example 1

[0120] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0121] Conventional information collection and management systems required a huge amount of manual work to digitize paper documents, resulting in a lack of efficiency. Furthermore, manual data collection and management entailed the risk of input errors and data leaks, making it unreliable. Furthermore, there were insufficient methods for accurately digitizing paper documents and managing them electronically.

[0122] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0123] In this invention, the server includes means for a user to take an image of a document with an image input device and upload the image to a data processing device, means for the data processing device to receive the image and extract character data from the image using optical character recognition means, means for the data processing device to analyze the extracted character data and the layout of the image and automatically generate a digital input form, means for transmitting the generated digital input form to a user interface for the user and displaying it on the user interface, and means for transmitting data entered by the user on the user interface to the data processing device and storing it in a database, thereby enabling rapid and accurate digitization of paper documents and efficient information collection and management.

[0124] An "image input device" is a device with a camera function, including a smartphone or tablet terminal.

[0125] "Document" refers to information recorded on paper, including bills, receipts, medical records, etc.

[0126] A "data processing device" is a computer system for processing and analyzing image data and text data.

[0127] "Optical character recognition" means software or hardware technology for converting character information in an image into digital text, such as an OCR engine.

[0128] "Character data" is character information in digital form extracted by optical character recognition means.

[0129] "Layout" refers to the position information of character data within an image, including the layout of a document and the position of fields.

[0130] A "digital input form" is an electronic input screen generated based on analyzed character data and layout information.

[0131] A "user interface" refers to the display elements of a terminal screen or application that a user operates and uses to input data.

[0132] "Database" means a structured computer system for storing and managing processed data, including relational database systems.

[0133] "Encryption" means a technique for securely transforming data for the purpose of protecting the data during transmission.

[0134] This invention is a system that digitizes paper documents, eliminates the need for manual data entry, and enables efficient and accurate information collection. This system uses the following hardware and software:

[0135] This system starts with the user taking a photo of a paper document using an image input device (such as a smartphone or tablet), allowing the user to capture paper documents such as invoices, receipts, and medical records as image data.

[0136] Next, the user uploads the captured image to the data processing device. Using a dedicated application, the user selects an image and taps the upload button. The application then sends the image data to the server via an HTTP POST request.

[0137] The server first receives the image data and temporarily stores it. Then, the server launches an OCR engine (e.g., Tesseract) to extract the text data from the image. Specifically, it calls the OCR engine's API, reads the image file, and converts the text data into a digital format. For example, to use Tesseract OCR, run a command like "tesseract image.png output -l eng --psm 1."

[0138] The server then performs layout analysis based on the positional information of the character data. This analysis identifies the position of each text field and automatically generates a digital input form that is displayed in a format similar to the layout of the original paper document.

[0139] The server sends this digital input form to the user's device. Specifically, it is encoded in HTML or JSON format and sent as an HTTP response. The user's device receives this data and displays it on the application.

[0140] The user enters the required data into the digital input form displayed on the terminal, such as the invoice item name, quantity, amount, etc. After completion, the user presses the "Submit" button to send the data to the server, which then sends the data back to the server via an HTTP POST request.

[0141] The server stores the received data in a database using a relational database management system (e.g., MySQL or PostgreSQL) to store and manage the data accurately and efficiently. For example, it executes an INSERT statement to insert the data into the database.

[0142] Specific examples

[0143] Retailer Use

[0144] For example, this system is very useful for retailers who want to digitize their daily sales data. A retailer's employee takes a photo of each customer's receipt with their smartphone and uploads the image to a server using an application. The server analyzes the image and generates a digital receipt entry form. The employee enters the data on the device and presses the "send" button to send the data to the server. The server stores the received data in a database, making it easy to digitize sales data.

[0145] Use of medical facilities

[0146] In medical facilities, patient medical records are often kept on paper. By using this system, medical records are digitized and managed efficiently. Medical staff take photos of paper medical records and upload them to the system using a dedicated application. The server analyzes the images and generates a digital medical record entry form. Staff enter the necessary data into the form on their device, press the "send" button, and send it to the server, thereby advancing the digitization of medical records.

[0147] Prompt Sentence Examples

[0148] "Please explain the system a retailer uses to digitize sales data. The system consists of image capture, upload, OCR analysis, input form generation, data entry, and storage."

[0149] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0150] Step 1: Take an image

[0151] The user takes an image of a paper document using an image input device (smartphone or tablet). The user launches a camera app, adjusts the camera position so that the entire document fits in the image, and presses the capture button. The input for this step is the paper document, and the output is image data (JPEG or PNG format).

[0152] Step 2: Upload your images to the system

[0153] The user launches the dedicated application, selects the captured image, and taps the upload button. The application sends the selected image data to the server as an HTTP POST request. The input of this step is the image data, and the output is the image data sent to the server.

[0154] Step 3: Receive and save the image

[0155] The server receives the image data sent from the user's device and temporarily stores it. Specifically, the server receives the image data using XMLHttpRequest or Fetch API and stores it in a local temporary directory. The input to this step is the image data as an HTTP POST request, and the output is an image file stored in the server's temporary directory.

[0156] Step 4: Perform Optical Character Recognition (OCR)

[0157] The server invokes an OCR engine (e.g., Tesseract) on the stored image. Specifically, it calls the OCR engine's API to analyze the image file and extract the text data from the image. For example, when using Tesseract, execute a command like "tesseract image.png output -l eng --psm 1". The input for this step is the image file, and the output is digital text data.

[0158] Step 5: Perform layout analysis

[0159] The server performs layout analysis based on the positional information of the character data extracted by OCR. Specifically, it analyzes the coordinate information of each text field and executes an algorithm to identify the document layout. The input to this step is the character data and its positional information, and the output is a digital input form based on the layout information.

[0160] Step 6: Generate a digital input form

[0161] The server generates a digital input form based on the layout analysis results. Specifically, it dynamically generates a form template and embeds the analysis results. The input for this step is layout information and character data, and the output is the HTML or JSON data of the digital input form.

[0162] Step 7: Submitting the digital input form

[0163] The server sends the generated digital input form to the user's device. Specifically, it sends data encoded in HTML or JSON format as an HTTP response. The input of this step is the digital input form data, and the output is the digital input form displayed on the user's device.

[0164] Step 8: Enter your data

[0165] The user enters data into a digital input form displayed on the terminal display. Specifically, data is entered into form elements such as text boxes and drop-down lists. The input of this step is the user's input data, and the output is the data entered into the form.

[0166] Step 9: Sending Data

[0167] After the user has completed entering data, they click the "Submit" button to send the input data to the server. Specifically, the form data is sent to the server as an HTTP POST request. The input of this step is the data entered by the user, and the output is the data sent to the server.

[0168] Step 10: Save your data

[0169] The server stores the received data in a database. Specifically, it uses a relational database management system (e.g., MySQL or PostgreSQL) to add records to the database by executing SQL INSERT statements. The input to this step is the data sent to the server, and the output is the records stored in the database.

[0170] (Application example 1)

[0171] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0172] When digitizing paper documents, manual data entry is extremely time-consuming and labor-intensive, and prone to human error. In addition, factories need to manage a wide range of paper documents, making efficient digitization a necessity. Furthermore, mechanical methods for converting paper data into digital format are not being used effectively.

[0173] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0174] In this invention, the server includes means for a user to photograph an image of a paper document and upload the image to the system, means for the server to receive the image and extract text data from the image using optical character recognition means, means for the server to analyze the extracted text data and the layout of the image and automatically generate a digital input form, means for sending the generated digital input form to the user's terminal and displaying it on the terminal, means for sending data entered by the user on the terminal to the server and storing it in a database, means for a robot to photograph a paper document and upload the image to the server, means for the robot to display the generated digital input form and for an operator to enter data, and means for sending the entered data to the server and storing it in a database, thereby enabling efficient and accurate digitization of paper documents.

[0175] "User" means a person or entity that uses the System to digitize paper documents.

[0176] "Paper documents" refers to documents or documents printed or recorded on paper, such as invoices, receipts, manuals, and checklists.

[0177] An "image" is digital visual information captured by a camera of a paper document.

[0178] In the present invention, the term "system" refers to a set of processes and devices for digitizing paper documents.

[0179] A "server" is a computing resource that analyzes received images, generates digital input forms, and stores data.

[0180] "Optical character recognition" is a technology for analyzing text within an image and converting it into digital character data.

[0181] A "digital input form" is a digital form into which a user can input data, constructed based on extracted text data and layout analysis.

[0182] A "robot" is a mechanical device that automatically photographs paper documents in factories and other work sites and uploads the images to the system.

[0183] A "database" is a system that stores input data and allows it to be managed and searched as needed.

[0184] A "worker" is a person whose role is to operate a robot or enter data into a digital input form.

[0185] MODE FOR CARRYING OUT THE INVENTION

[0186] This invention is a system that digitizes paper documents and enables efficient and accurate information collection. The system starts when a user takes a photo of a paper document and uploads the image to a server using a robot or terminal.

[0187] The server first receives the image data and extracts the text data from the image using optical character recognition (OCR). Specifically, an OCR library such as Tesseract is used. The extracted text data and the image layout are then analyzed to generate a digital input form. OpenCV is used to preprocess the image, removing noise and detecting contours, enabling accurate text extraction.

[0188] The generated digital input form is sent to the user's terminal or robot and displayed. The user or worker enters data on these terminals or devices, and the data is then sent back to the server and stored in a database. This makes data entry from paper more efficient and reduces input errors.

[0189] This system is particularly effective in factories. The process of factory robots automatically photographing and digitizing paper checklists dramatically improves on-site data management. Furthermore, by analyzing the layout of the input form, a digital form is created that closely matches the format of the original paper document, allowing workers to enter data without any discomfort.

[0190] A concrete example would be the use of a checklist in a factory. For example, if there is a checklist like this:

[0191] Machine name: XYZ product

[0192] Check item: Oil level

[0193] Inspection result: Pass Fail

[0194] The robot takes a photo of this checklist and uploads it to the system. The text is analyzed by Tesseract and sent to the server, where an input form is generated based on the layout analysis.

[0195] Examples of prompts include:

[0196] Generate a digital input form from the following text:

[0197] Machine name: XYZ product

[0198] Check item: Oil level

[0199] Inspection result: Pass Fail

[0200] The generated form should have the following format:

[0201] {

[0202] "formFields": [

[0203] {"label": "machine name", "type": "text"},

[0204] {"label": "Inspection items", "type": "text"},

[0205] {"label": "Inspection result", "type": "radio", "options": ["Pass", "Fail"]}

[0206] ]

[0207] }

[0208] Using these prompts, accurate digital input forms are automatically generated. The data is then stored in a database for further processing and analysis. This system allows for the rapid and accurate digitization of paper-based data.

[0209] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0210] Step 1:

[0211] A user takes an image of a paper document.

[0212] Input: Paper documents

[0213] Output: Captured image data (e.g., JPEG image file)

[0214] Specific operation: The user uses the camera on their smartphone or tablet to take a picture of the checklist or manual.

[0215] Step 2:

[0216] The robot uploads the captured images to the system.

[0217] Input: Photographed image data

[0218] Output: Image data sent to the server

[0219] Specific operation: Images taken by the camera mounted on the robot are transferred to the built-in control unit and uploaded to a server via the network.

[0220] Step 3:

[0221] The server receives the uploaded images and temporarily stores them in a database.

[0222] Input: Uploaded image data

[0223] Output: Temporarily saved image data

[0224] Specific operation: The server receives the HTTP request and saves the image data in a designated folder for temporary storage.

[0225] Step 4:

[0226] The server performs OCR (optical character recognition) and extracts text data from the image data.

[0227] Input: Temporarily saved image data

[0228] Output: Extracted text data

[0229] How it works: The server uses an OCR library such as Tesseract to analyze the letters and numbers in the image and convert them into digital text.

[0230] Step 5:

[0231] The server performs layout analysis and generates a digital input form.

[0232] Input: Extracted text data and image layout information

[0233] Output: Generated digital input form (e.g. JSON format)

[0234] Specific operation: The server uses image processing libraries such as OpenCV to analyze the positional information of the text and generate a digital input form that closely resembles the layout of paper media.

[0235] Step 6:

[0236] The server sends the generated digital input form to the terminal or robot.

[0237] Input: Generated digital input form

[0238] Output: Submitted digital input form

[0239] Specific operation: The server sends the generated form to the terminal or robot as an HTTP response.

[0240] Step 7:

[0241] Users or workers enter data into a digital input form displayed on a terminal or robot display.

[0242] Input: Digital input form

[0243] Output: The data entered (e.g., the value of a form field)

[0244] Specific operation: The user or worker uses the touch panel or keyboard of the terminal or robot to enter the required data into the displayed form.

[0245] Step 8:

[0246] The terminal or robot sends the entered data to the server.

[0247] Input: Data entered

[0248] Output: Data sent to the server

[0249] Specific operation: When a user or worker presses the "Send" button, the entered data is sent to the server as an HTTP request.

[0250] Step 9:

[0251] The server stores the received data in a database.

[0252] Input: Data sent to the server

[0253] Output: Data stored in the database

[0254] What happens: The server converts the received data into an appropriate format and inserts it into a database, where it can be used for later analysis, reporting, etc.

[0255] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0256] The present invention is a system that digitizes paper documents, eliminates the need for manual data entry, and enables efficient and accurate information collection, and further includes a means for recognizing the user's emotions and reflecting the results in the system. Specific embodiments of the various means are described below.

[0257] Program processing overview

[0258] Image upload

[0259] 1. The user takes a picture on the device

[0260] Users use the camera on their smartphone or tablet to take a picture of a paper document (such as an invoice or receipt) using either a dedicated camera app or the device's standard camera app.

[0261] 2. User uploads image to the system

[0262] The user launches a dedicated application, selects the captured image, and presses the "upload" button to send the image to the system.

[0263] Image processing, analysis and emotion recognition

[0264] 3. The server receives the image

[0265] The server receives the image sent from the user's device, temporarily stores it in storage, and returns a response confirming receipt to the device.

[0266] 4. The server runs the OCR engine

[0267] The server runs an optical character recognition (OCR) engine on the received image to extract text data from the image. For example, if the server uses the Tesseract OCR engine, it calls its API to process the data.

[0268] 5. The server performs layout analysis

[0269] The server analyzes the layout of the document based on the location information of the text data, and as a result, the digital input form is reproduced with the same layout as the paper document.

[0270] 6. The server runs the emotion engine

[0271] The server analyzes the user's image and input actions and activates an emotion engine that recognizes the user's emotions. For example, it uses an emotion recognition API to analyze emotions from the user's facial expressions and input speed.

[0272] Input form generation and display

[0273] 7. The server generates the digital input form

[0274] The server generates a digital input form based on the analysis results, which has the same layout (position of input fields, item names, etc.) as the original paper document.

[0275] 8. The server encodes the input form

[0276] The server encodes the generated digital input form in a format such as JSON or HTML and prepares it to be sent to the user's device.

[0277] 9. The server sends the input form to the device

[0278] The server transmits the encoded digital input form data to the user's terminal, including the transmitted data as a response.

[0279] 10. The device receives the input form

[0280] The terminal receives and decodes the digital input form data sent from the server, thereby preparing to display the digital input form on the terminal.

[0281] 11. The device displays the input form

[0282] The terminal displays the received digital input form on the screen so that the user can fill it in. The user interface is designed to be intuitive.

[0283] Data entry and transmission and emotional feedback

[0284] 12. User enters data into input form

[0285] The user inputs necessary data into the digital input form displayed on the terminal, such as product name, quantity, and price.

[0286] 13. Feedback from the Emotion Engine

[0287] The emotion engine dynamically changes the design and message as you type, reflecting your emotions. For example, if you're feeling stressed, it will present a relaxing message or color tone.

[0288] 14. Save or submit user-entered data

[0289] After completing the data entry, the user presses the "Save" or "Send" button. If the "Save" button is pressed, the data is temporarily saved on the terminal, and if the "Send" button is pressed, the data is sent to the server.

[0290] 15. The device sends the input data to the server

[0291] When the terminal transmits the input data to the server via an HTTP request, the terminal encrypts the data as necessary.

[0292] 16. The server receives and stores the data

[0293] The server validates the data received from the devices and stores it in a database for further processing and report generation.

[0294] 17. The server performs data analysis and report generation

[0295] The server performs statistical processing and generates reports based on the stored data, such as aggregating sales data and generating monthly reports.

[0296] Specific examples

[0297] Retailer Use

[0298] For example, this system would be extremely useful for retailers to easily digitize their daily sales data. Retailers' staff members take photos of each customer's receipts with their smartphones and upload them. The server analyzes the images and generates a digital receipt entry form. The staff member enters the data on the device and stores it on the server, making it easy to digitize sales data. In addition, the emotion engine detects the staff member's stress and provides feedback to help them relax.

[0299] Use of medical facilities

[0300] In medical facilities, patient medical records are often kept on paper. By using this system, medical records are digitized and managed efficiently. Medical staff take photos of paper medical records and upload them to the system. The server analyzes the images and generates a digital medical record entry form. Staff enter the necessary data into the form and save it on the server, further increasing the digitization of medical records. An emotion engine detects staff fatigue and provides refreshing messages, improving work efficiency and comfort.

[0301] These processes enable paper documents to be digitized quickly and accurately, improving work efficiency and data management. Furthermore, the use of an emotion engine is expected to reduce user stress and improve work efficiency.

[0302] The processing flow will be explained below.

[0303] Step 1:

[0304] The user takes a picture on the device

[0305] Users use the camera on their smartphone or tablet to take a picture of a paper document (such as an invoice or receipt) using either a dedicated camera app or the device's standard camera app.

[0306] Step 2:

[0307] User uploads an image to the system

[0308] The user launches a dedicated application, selects the captured image, and presses the "upload" button to send the image to the system.

[0309] Step 3:

[0310] The device sends the image to the server

[0311] The device uses an HTTP request (such as a POST request) to send the captured image to the server, compressing or encrypting the image data as necessary.

[0312] Step 4:

[0313] The server receives the image

[0314] The server receives the image sent by the user, temporarily stores it in storage, and returns a response confirming receipt to the terminal.

[0315] Step 5:

[0316] The server runs the OCR engine

[0317] The server runs an optical character recognition (OCR) engine on the received image to extract text data from the image. For example, if the server uses the Tesseract OCR engine, it calls its API to process the data.

[0318] Step 6:

[0319] The server performs the layout analysis

[0320] The server analyzes the layout of the document based on the location information of the text data, and as a result, the digital input form is reproduced with the same layout as the paper document.

[0321] Step 7:

[0322] The server runs the emotion engine

[0323] The server analyzes the user's image and input actions and activates an emotion engine that recognizes the user's emotions. For example, it uses an emotion recognition API to analyze emotions from the user's facial expressions and input speed.

[0324] Step 8:

[0325] The server generates the digital input form

[0326] The server generates a digital input form based on the analysis results, which has the same layout (position of input fields, item names, etc.) as the original paper document.

[0327] Step 9:

[0328] The server encodes the input form

[0329] The server encodes the generated digital input form in a format such as JSON or HTML and prepares it to be sent to the user's device.

[0330] Step 10:

[0331] The server sends the input form to the terminal

[0332] The server transmits the encoded digital input form data to the user's terminal, including the transmitted data as a response.

[0333] Step 11:

[0334] The device receives the input form

[0335] The terminal receives and decodes the digital input form data sent from the server, thereby preparing to display the digital input form on the terminal.

[0336] Step 12:

[0337] The device displays the input form

[0338] The terminal displays the received digital input form on the screen so that the user can fill it in. The user interface is designed to be intuitive.

[0339] Step 13:

[0340] The user enters data into the input form

[0341] The user inputs necessary data into the digital input form displayed on the terminal, such as product name, quantity, and price.

[0342] Step 14:

[0343] Feedback from the Emotion Engine

[0344] The server's emotion engine monitors the user's facial expressions and behavior while they are typing and evaluates them in real time. For example, if the user is feeling stressed, a relaxing message or color tone will be dynamically displayed on the input form.

[0345] Step 15:

[0346] User saves or submits input data

[0347] After completing the data entry, the user presses the "Save" or "Send" button. When the "Save" button is pressed, the data is temporarily saved on the terminal, and when the "Send" button is pressed, the data is sent to the server.

[0348] Step 16:

[0349] The device sends the input data to the server

[0350] The terminal sends the input data to the server via an HTTP request, encrypting the data if necessary.

[0351] Step 17:

[0352] The server receives and stores the data

[0353] The server verifies the data received from the devices and stores it in a database for further processing and report generation.

[0354] Step 18:

[0355] The server performs data analysis and report generation

[0356] The server performs statistical processing and generates reports based on the stored data, such as aggregating sales data and generating monthly reports.

[0357] Through these steps, paper documents can be digitized quickly and accurately, improving work efficiency and data management. Furthermore, the use of an emotion engine is expected to reduce user stress and improve work efficiency.

[0358] Example 2

[0359] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0360] Digitizing paper documents requires manual data entry, which is time-consuming and labor-intensive. Furthermore, traditional systems often have difficulty accurately reproducing the layout of documents. Furthermore, there is no feedback based on the user's emotions or state, resulting in a lack of consideration for user stress and work efficiency.

[0361] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for extracting text data from an image using optical character recognition means, means for automatically generating a digital input form by analyzing the extracted text data and the layout of the image, and means for analyzing the user's image and input actions, recognizing the user's emotions, and providing feedback. This makes it possible to efficiently and accurately digitize paper documents, further reducing user stress and improving work efficiency.

[0362] "User" means any person or legal entity that uses the System to digitize paper documents.

[0363] A "paper document" is a document containing information printed or handwritten on paper.

[0364] "Means for taking images" refers to a method or device for taking images of paper documents using a device equipped with a camera function (such as a smartphone or tablet).

[0365] "System" means a set of devices and programs, including servers and terminals, for digitizing paper documents and processing and storing data.

[0366] A "server" is a computer that receives images, processes data, and stores it.

[0367] "Optical character recognition" refers to technology or equipment that analyzes character information in an image and extracts it as text data. An example of this is an OCR (optical character recognition) engine.

[0368] "Text data" refers to character information extracted by optical character recognition means.

[0369] A "layout analysis means" is a technique or device that analyzes the structure and layout of a document based on the positional information of text data within an image.

[0370] A "digital input form" is an input form that is reproduced in a digital environment based on a paper document.

[0371] "Means for providing feedback" refers to technologies or devices that provide appropriate responses or advice in real time based on the user's emotions and input actions.

[0372] "Encryption means" refers to a technique or device that encrypts data to ensure data security.

[0373] A "database" is a storage device or system for storing and managing data received by a server.

[0374] "Emotion recognition means" refers to technology or devices that analyze and evaluate a user's emotional state based on the user's image and input actions.

[0375] This system aims to digitize paper documents, improve work efficiency, and organize data accurately. Furthermore, it analyzes user emotions and provides feedback to improve usability. This system operates in cooperation with users, terminals, and servers.

[0376] Hardware and software used

[0377] Hardware: Smartphones, tablets, and server computers used by users

[0378] Software: Camera app (dedicated app or device-specific app), OCR engine (e.g., Tesseract), emotion recognition API, database system

[0379] Program processing overview

[0380] 1. The user takes a picture on the device

[0381] The user uses the camera on their smartphone or tablet to take a picture of a paper document (e.g., an invoice or receipt) using a dedicated app or the device's built-in camera app.

[0382] 2. User uploads image to the system

[0383] The user launches a dedicated application, selects the captured image, and presses the "upload" button to send the image to the system.

[0384] 3. The server receives the image

[0385] The server receives the image sent from the user's device, temporarily stores it in storage, and returns a response confirming receipt to the device.

[0386] 4. The server runs the OCR engine

[0387] The server uses an optical character recognition (OCR) engine to extract text data from the image. Specifically, it calls the API of the Tesseract OCR engine to obtain the character information in the image as text data.

[0388] 5. The server performs layout analysis

[0389] The server analyzes the layout based on the location information of the extracted text data and generates a digital input form, which creates a digital form that retains the same layout as the paper document.

[0390] 6. The server runs the emotion engine

[0391] The server analyzes the user's image and input actions to activate an emotion engine that recognizes emotions. Specifically, it uses an emotion recognition API to analyze the user's emotions based on their facial expressions and input speed, and provides feedback.

[0392] 7. The server generates and submits the digital input form

[0393] The server generates a digital input form based on the analysis results, encodes this form in JSON or HTML format, and sends it to the user's device.

[0394] 8. The device receives and displays the input form

[0395] The terminal receives the digital input form data sent from the server, decodes it, and displays it, allowing the user to enter the required data into the digital input form.

[0396] 9. The device sends the input data to the server

[0397] After the user has completed the data entry, the data is sent from the terminal to the server, where it is encrypted if necessary.

[0398] 10. The server receives and stores the data

[0399] The server validates the received data and stores it securely in a database for later statistical processing and report generation.

[0400] 11. The server performs data analysis and report generation

[0401] The server performs statistical processing and generates reports based on the stored data, such as compiling sales data and generating monthly reports.

[0402] Specific examples

[0403] For retailers

[0404] This system is extremely useful for retailers digitizing their daily sales data. When a salesperson takes a photo of a customer's receipt with their smartphone and uploads it, the server analyzes the image and generates a digital receipt entry form. The salesperson enters the data on the device and sends it to the server. The system uses an emotion engine to detect the salesperson's stress and provides a relaxing message.

[0405] For medical facilities

[0406] Medical facilities use this system to digitize paper medical records. When medical staff take a photo of a paper medical record and upload it to the system, the server analyzes the image and generates a digital input form. The staff enters data into the form and sends it to the server. An emotion engine detects staff fatigue and displays a refresh message. This allows for efficient management of medical records and improves work comfort.

[0407] In this way, the system of the present invention not only efficiently and accurately digitizes paper documents, but also provides technical means for reducing user stress and improving work efficiency.

[0408] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0409] Step 1:

[0410] The user takes a photo of a paper document

[0411] Input: Paper documents (e.g. invoices, receipts)

[0412] How it works: The user uses the built-in camera on their smartphone or tablet to take a photo of a paper document using a dedicated app or the device's standard camera app.

[0413] Output: Captured image data

[0414] Step 2:

[0415] A user uploads an image to the system

[0416] Input: Captured image data

[0417] How it works: The user launches the dedicated application, selects the captured image, and presses the "Upload" button, which sends the image data to the server via the Internet.

[0418] Output: Image data sent to the server

[0419] Step 3:

[0420] The server receives the image

[0421] Input: Image data sent by the user

[0422] Operation: The server temporarily stores the received image data in storage and returns a response confirming receipt to the user device.

[0423] Output: Image data stored in storage, receipt confirmation response

[0424] Step 4:

[0425] The server runs the OCR engine

[0426] Input: Received and stored image data

[0427] How it works: The server uses an optical character recognition (OCR) engine (e.g., Tesseract) to extract character information from the image data as text data.

[0428] Output: Extracted text data

[0429] Step 5:

[0430] The server performs the layout analysis

[0431] Input: Extracted text and image data

[0432] How it works: The server analyzes the location information of the text data and the layout of the images, and automatically generates a digital input form based on the layout of the paper medium.

[0433] Output: The generated digital input form

[0434] Step 6:

[0435] The server runs the emotion engine

[0436] Input: User images, input data, and action data

[0437] How it works: The server uses the emotion recognition API to analyze the user's facial expressions and input speed to recognize their emotions. Based on the analysis results, it generates feedback content for the user.

[0438] Output: User sentiment analysis results, feedback content

[0439] Step 7:

[0440] The server generates and submits the digital input form

[0441] Input: Digital input form, feedback content

[0442] How it works: The server encodes the generated digital input form in JSON or HTML format and sends it to the user's device, optionally incorporating feedback into the submitted data.

[0443] Output: Encoded digital input form data, feedback data

[0444] Step 8:

[0445] The device receives and displays the input form.

[0446] Input: Digital input form data sent from the server

[0447] Operation: The user terminal receives the image data, decodes it, and displays an input form. The user can then enter the required data into the form.

[0448] Output: Digital input form displayed on the terminal

[0449] Step 9:

[0450] The user enters data into the form

[0451] Input: Digital input forms, data entered by users

[0452] How it works: The user enters data such as product name, quantity, and price into a digital input form displayed on the terminal.

[0453] Output: Data entered by the user

[0454] Step 10:

[0455] The device sends the input data to the server

[0456] Input: Data entered by the user

[0457] How it works: When a user presses the "Save" or "Submit" button, the device sends the input data to the server via an HTTP request, encrypting the data if necessary.

[0458] Output: Input data sent to the server

[0459] Step 11:

[0460] The server receives and stores the data

[0461] Input: Input data sent from the terminal

[0462] Operation: The server stores the received data in a database for later statistical processing and report generation.

[0463] Output: Data stored in the database

[0464] Step 12:

[0465] The server performs data analysis and report generation.

[0466] Input: Data stored in the database

[0467] How it works: The server processes the stored data and generates statistics and reports, such as aggregating sales data and generating monthly reports.

[0468] Output: easy-to-read statistics and reports

[0469] (Application example 2)

[0470] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0471] In food delivery operations, digitizing handwritten order forms and delivery confirmations has become a problem, resulting in time, mistakes, and stress for delivery workers. Furthermore, managing paper documents is complicated, creating a need for efficient data management. Furthermore, there is a lack of methods to reduce the workload and manage stress for delivery workers.

[0472] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for a user to take an image of a paper document and upload the image to the system; means for the server to receive the image and extract text data from the image using optical character recognition means; means for the server to analyze the extracted text data and the image layout and automatically generate a digital input form; means for the server to send the generated digital input form to the user's terminal and display it on the terminal; means for the server to send data entered by the user on the terminal and store it in a database; and means for the server to recognize the user's emotions using image analysis means and provide feedback based on the emotions to the terminal. This allows delivery personnel to efficiently and accurately digitize paper documents and manage stress in real time.

[0473] Definitions of important words

[0474] "User" means a person or organization that uses a system to digitize paper documents.

[0475] A "paper document" is a document containing information printed or handwritten on paper.

[0476] An "image" is visual data captured using a device such as a camera.

[0477] "Upload" is the act of sending data from a user's device to a server.

[0478] "System" refers to a set of technological means, including computers and software.

[0479] A "server" is a computer system that centrally receives, processes, and stores data.

[0480] "Optical character recognition" is a technology that analyzes text within an image and converts it into digital data.

[0481] "Text data" is digital information expressed as a string of characters.

[0482] "Layout" refers to information about the arrangement of text, images, etc. in a paper document.

[0483] "Digital input form" means an electronic form for entering data from a digitized document.

[0484] A "terminal" is an electronic device operated by a user, such as a smartphone or tablet.

[0485] "Image analysis means" is a technique for extracting specific information from an image.

[0486] "Emotion recognition" is a technology that analyzes emotions from a user's facial expressions and behavior.

[0487] "Feedback" refers to the information or response that a system provides to a user.

[0488] A "database" is a system for efficiently storing and managing digital data.

[0489] Description (Mode for carrying out the invention)

[0490] The present invention provides a system that allows users to digitize paper documents and manage information efficiently and accurately. Specific embodiments of the present invention will be described below.

[0491] Hardware and Software Configuration

[0492] Hardware:

[0493] Device: An electronic device such as a smartphone or tablet.

[0494] Server: A computer system that receives, processes, and stores data.

[0495] software:

[0496] Flask: Implementing server programs using a web framework.

[0497] PaddleOCR: An optical character recognition engine for extracting text from images.

[0498] FER: Analyzes emotions from facial expressions in images through an emotion recognition engine.

[0499] OpenCV: Use the image processing library to analyze and process images.

[0500] System Operation

[0501] The server performs the following process.

[0502] 1. Receiving and saving images: This receives images of paper documents taken by users on their devices and temporarily saves them in storage. This is implemented using the Flask framework.

[0503] 2. Text extraction using OCR: PaddleOCR is used to extract text data from images. The OCR engine processes the received image and returns the extracted text data to the server.

[0504] 3. Layout analysis: The text data extracted by OCR is analyzed to recreate the same layout as the original paper document. This processing is performed on the server side, allowing the generation of a digital input form.

[0505] 4. Emotion Recognition: FER is used to analyze the user's emotions. Based on facial expressions in the image and input speed, emotions are analyzed and feedback corresponding to the emotion is provided to the device.

[0506] 5. Sending and displaying the digital input form: After the analysis is complete, the generated digital input form is encoded in JSON or HTML format and sent to the user's device, where it displays the received data and allows the user to enter information.

[0507] Specific examples

[0508] For example, this system is extremely useful when food delivery workers need to digitize handwritten order forms. The delivery worker takes a photo of the order form with their smartphone and uploads it to a server using a dedicated application. The server receives the image, extracts text data using OCR, and generates a digital input form by analyzing the layout. The generated digital input form is then sent to a terminal, where the delivery worker can enter the necessary information. In addition, an emotion recognition engine can provide relaxing messages or feedback if the delivery worker's stress level is high.

[0509] Example prompt sentence:

[0510] "Please recognize characters in an image and convert them into digital text. Also, please analyze the emotions of faces in the image and return the results in JSON format."

[0511] In this way, paper documents can be digitized quickly and accurately, reducing the workload of delivery personnel and improving work efficiency.

[0512] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0513] Program processing steps

[0514] Step 1:

[0515] The user (delivery person) takes a picture of the paper order form using the camera on their smartphone. The image is temporarily saved in the device's storage.

[0516] Input: Paper purchase order

[0517] Output: Image file saved on device

[0518] Step 2:

[0519] The user launches the dedicated application, selects the image they have taken, and presses the "upload" button, which sends the image to the server.

[0520] Input: Image file saved on your smartphone

[0521] Output: Image data sent to the server

[0522] Step 3:

[0523] The server receives the image, temporarily stores it in storage, and returns a reception confirmation response to the device.

[0524] Input: Image data sent by the user

[0525] Output: Image file saved on the server and a receipt confirmation response to the device

[0526] Step 4:

[0527] The server runs an OCR engine (PaddleOCR) to extract text data from the received images.

[0528] Input: Image file stored on the server

[0529] Output: Extracted text data

[0530] Step 5:

[0531] The server analyzes the layout and generates a digital input form based on the positional information of the text data, recreating the same layout (position of input fields, item names, etc.) as the original paper document.

[0532] Input: Text data extracted by OCR

[0533] Output: Digital input form data

[0534] Step 6:

[0535] The server runs an emotion recognition engine (FER) to analyze the user's emotions from their facial expressions and input actions, and generates appropriate feedback and sends it to the user's device.

[0536] Input: Received image and user input action data

[0537] Output: User sentiment analysis results and feedback

[0538] Step 7:

[0539] The server encodes the generated digital input form in JSON or HTML format and sends it to the user's device. The device then decodes the received digital input form data and displays it on the screen.

[0540] Input: Digital input form data

[0541] Output: A digital input form displayed on the user's device

[0542] Step 8:

[0543] The user enters the necessary data into the digital input form displayed on the terminal, such as product information, quantity, and price.

[0544] Input: User-entered data

[0545] Output: Input digital data

[0546] Step 9:

[0547] The user presses a button to save or send the input data. If the save button is pressed, the data is temporarily saved on the terminal, and if the send button is pressed, the data is sent to the server.

[0548] Input: User input data and button operations

[0549] Output: Data stored on the device or sent to a server

[0550] Step 10:

[0551] The server stores the received data in a database and performs statistical processing and generates reports as needed.

[0552] Input: Transmitted digital data

[0553] Output: Data stored in the database and generated reports

[0554] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0555] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0556] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0557] [Second embodiment]

[0558] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0559] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0560] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0561] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0562] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0563] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0564] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0565] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0566] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0567] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0568] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0569] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0570] The present invention is a system that digitizes paper documents, eliminates the need for manual data entry, and enables efficient and accurate information collection. This system functions as follows.

[0571] Program processing overview

[0572] Image upload

[0573] 1. The user takes a picture on the device

[0574] Users use the camera on their smartphone or tablet to take a picture of a paper document, such as an invoice or receipt.

[0575] 2. User uploads image to the system

[0576] Users upload the images they have taken to the system through a dedicated application. To do so, they press the "upload" button to send the image files to the server.

[0577] Image processing and analysis

[0578] 3. The server receives the image

[0579] The server receives the image sent from the user's device and temporarily stores it for further processing.

[0580] 4. The server performs optical character recognition (OCR)

[0581] The server runs an OCR engine on the received image, analyzes the text information in the image, and extracts the text data in a digital format, possibly using an OCR library such as Tesseract.

[0582] 5. The server performs layout analysis

[0583] The server analyzes the layout based on the location information of the text data and identifies the structure of the input form, allowing the digital input form to be reproduced with the same layout as the paper document.

[0584] Input form generation

[0585] 6. The server generates the digital input form

[0586] The server generates a template based on the analysis results and creates an input form with a layout similar to the original document.

[0587] 7. The server sends the input form to the device

[0588] The server sends the generated digital input form to the user's device, where it is encoded in JSON or HTML and displayed on the user's device.

[0589] Data entry and saving

[0590] 8. The user enters data on the device

[0591] The user enters data into a digital input form displayed on the terminal, such as invoice item names, quantities, and amounts.

[0592] 9. User submits input data

[0593] After completing the data entry, the user presses the "Submit" button to send the entered data to the server.

[0594] 10. The server stores the data

[0595] The server stores the received data in a database and manages the necessary information for subsequent processing and report generation.

[0596] Specific examples

[0597] Retailer Use

[0598] For example, this system is very useful for retailers to easily digitize their daily sales data. Retailers' staff take photos of each customer's receipt with their smartphones and upload them. The server analyzes the images and generates a digital receipt entry form. The staff enters the data on the device and saves it on the server, making it easy to digitize sales data.

[0599] Use of medical facilities

[0600] In medical facilities, patient medical records are often kept on paper. By using this system, medical records are digitized and managed efficiently. Medical staff take photos of paper medical records and upload them to the system. The server analyzes the images and generates a digital medical record entry form. Staff enter the necessary data into the form and save it on the server, furthering the digitization of medical records.

[0601] These processes enable paper documents to be digitized quickly and accurately, resulting in more efficient operations and accurate data management.

[0602] The processing flow will be explained below.

[0603] Step 1:

[0604] The user takes a photo on the device

[0605] Users use the camera on their smartphone or tablet to take a picture of a paper document (such as an invoice or receipt) using either a dedicated camera app or the device's standard camera app.

[0606] Step 2:

[0607] User uploads an image to the system

[0608] The user launches a dedicated application, selects the captured image, and presses the "upload" button to send the image to the system.

[0609] Step 3:

[0610] The device sends the image to the server

[0611] The device uses an HTTP request (such as a POST request) to send the captured image to the server, compressing or encrypting the image data as necessary.

[0612] Step 4:

[0613] The server receives the image

[0614] The server receives the image sent by the user, temporarily stores it in storage, and returns a response confirming receipt to the terminal.

[0615] Step 5:

[0616] The server runs the OCR engine

[0617] The server runs an optical character recognition (OCR) engine on the received image to extract text data from the image. For example, if the server uses the Tesseract OCR engine, it calls its API to process the data.

[0618] Step 6:

[0619] The server performs the layout analysis

[0620] The server analyzes the document layout based on the location information of the extracted text data, in order to identify the structure of tables and the positions of input fields, and obtain data for generating a digital input form.

[0621] Step 7:

[0622] The server generates the digital input form

[0623] The server generates a digital input form based on the analysis results, which has the same layout (position of input fields, item names, etc.) as the original paper document.

[0624] Step 8:

[0625] The server encodes the input form

[0626] The server encodes the generated digital input form in a format such as JSON or HTML and prepares it to be sent to the user's device.

[0627] Step 9:

[0628] The server sends the input form to the terminal

[0629] The server transmits the encoded digital input form data to the user's terminal, including the transmitted data as a response.

[0630] Step 10:

[0631] The device receives the input form

[0632] The terminal receives and decodes the digital input form data sent from the server, thereby preparing to display the digital input form on the terminal.

[0633] Step 11:

[0634] The device displays the input form

[0635] The terminal displays the received digital input form on the screen so that the user can fill it in. The user interface is designed to be intuitive.

[0636] Step 12:

[0637] The user enters data into the input form

[0638] The user inputs necessary data into the digital input form displayed on the terminal, such as product name, quantity, and price.

[0639] Step 13:

[0640] User saves or submits input data

[0641] After completing the data entry, the user presses the "Save" or "Send" button. If the "Save" button is pressed, the data is temporarily saved on the terminal, and if the "Send" button is pressed, the data is sent to the server.

[0642] Step 14:

[0643] The device sends the input data to the server

[0644] The terminal sends the input data to the server via an HTTP request, encrypting the data if necessary.

[0645] Step 15:

[0646] The server receives and stores the data

[0647] The server validates the data received from the devices and stores it in a database for further processing and report generation.

[0648] Step 16:

[0649] The server performs data analysis and report generation

[0650] The server performs statistical processing and generates reports based on the stored data, such as aggregating sales data and generating monthly reports.

[0651] Through these steps, paper documents can be digitized quickly and accurately, resulting in more efficient operations and accurate data management.

[0652] Example 1

[0653] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0654] Conventional information collection and management systems required a huge amount of manual work to digitize paper documents, resulting in a lack of efficiency. Furthermore, manual data collection and management entailed the risk of input errors and data leaks, making it unreliable. Furthermore, there were insufficient methods for accurately digitizing paper documents and managing them electronically.

[0655] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0656] In this invention, the server includes means for a user to take an image of a document with an image input device and upload the image to a data processing device, means for the data processing device to receive the image and extract character data from the image using optical character recognition means, means for the data processing device to analyze the extracted character data and the layout of the image and automatically generate a digital input form, means for transmitting the generated digital input form to a user interface for the user and displaying it on the user interface, and means for transmitting data entered by the user on the user interface to the data processing device and storing it in a database, thereby enabling rapid and accurate digitization of paper documents and efficient information collection and management.

[0657] An "image input device" is a device with a camera function, including a smartphone or tablet terminal.

[0658] "Document" refers to information recorded on paper, including bills, receipts, medical records, etc.

[0659] A "data processing device" is a computer system for processing and analyzing image data and text data.

[0660] "Optical character recognition" means software or hardware technology for converting character information in an image into digital text, such as an OCR engine.

[0661] "Character data" is character information in digital form extracted by optical character recognition means.

[0662] "Layout" refers to the position information of character data within an image, including the layout of a document and the position of fields.

[0663] A "digital input form" is an electronic input screen generated based on analyzed character data and layout information.

[0664] A "user interface" refers to the display elements of a terminal screen or application that a user operates and uses to input data.

[0665] "Database" means a structured computer system for storing and managing processed data, including relational database systems.

[0666] "Encryption" means a technique for securely transforming data for the purpose of protecting the data during transmission.

[0667] This invention is a system that digitizes paper documents, eliminates the need for manual data entry, and enables efficient and accurate information collection. This system uses the following hardware and software:

[0668] This system starts with the user taking a photo of a paper document using an image input device (such as a smartphone or tablet), allowing the user to capture paper documents such as invoices, receipts, and medical records as image data.

[0669] Next, the user uploads the captured image to the data processing device. Using a dedicated application, the user selects an image and taps the upload button. The application then sends the image data to the server via an HTTP POST request.

[0670] The server first receives the image data and temporarily stores it. Then, the server launches an OCR engine (e.g., Tesseract) to extract the text data from the image. Specifically, it calls the OCR engine's API, reads the image file, and converts the text data into a digital format. For example, to use Tesseract OCR, run a command like "tesseract image.png output -l eng --psm 1."

[0671] The server then performs layout analysis based on the positional information of the character data. This analysis identifies the position of each text field and automatically generates a digital input form that is displayed in a format similar to the layout of the original paper document.

[0672] The server sends this digital input form to the user's device. Specifically, it is encoded in HTML or JSON format and sent as an HTTP response. The user's device receives this data and displays it on the application.

[0673] The user enters the required data into the digital input form displayed on the terminal, such as the invoice item name, quantity, amount, etc. After completion, the user presses the "Submit" button to send the data to the server, which then sends the data back to the server via an HTTP POST request.

[0674] The server stores the received data in a database using a relational database management system (e.g., MySQL or PostgreSQL) to store and manage the data accurately and efficiently. For example, it executes an INSERT statement to insert the data into the database.

[0675] Specific examples

[0676] Retailer Use

[0677] For example, this system is very useful for retailers who want to digitize their daily sales data. A retailer's employee takes a photo of each customer's receipt with their smartphone and uploads the image to a server using an application. The server analyzes the image and generates a digital receipt entry form. The employee enters the data on the device and presses the "send" button to send the data to the server. The server stores the received data in a database, making it easy to digitize sales data.

[0678] Use of medical facilities

[0679] In medical facilities, patient medical records are often kept on paper. By using this system, medical records are digitized and managed efficiently. Medical staff take photos of paper medical records and upload them to the system using a dedicated application. The server analyzes the images and generates a digital medical record entry form. Staff enter the necessary data into the form on their device, press the "send" button, and send it to the server, thereby advancing the digitization of medical records.

[0680] Prompt Sentence Examples

[0681] "Please explain the system a retailer uses to digitize sales data. The system consists of image capture, upload, OCR analysis, input form generation, data entry, and storage."

[0682] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0683] Step 1: Take an image

[0684] The user takes an image of a paper document using an image input device (smartphone or tablet). The user launches a camera app, adjusts the camera position so that the entire document fits in the image, and presses the capture button. The input for this step is the paper document, and the output is image data (JPEG or PNG format).

[0685] Step 2: Upload your images to the system

[0686] The user launches the dedicated application, selects the captured image, and taps the upload button. The application sends the selected image data to the server as an HTTP POST request. The input of this step is the image data, and the output is the image data sent to the server.

[0687] Step 3: Receive and save the image

[0688] The server receives the image data sent from the user's device and temporarily stores it. Specifically, the server receives the image data using XMLHttpRequest or Fetch API and stores it in a local temporary directory. The input to this step is the image data as an HTTP POST request, and the output is an image file stored in the server's temporary directory.

[0689] Step 4: Perform Optical Character Recognition (OCR)

[0690] The server invokes an OCR engine (e.g., Tesseract) on the stored image. Specifically, it calls the OCR engine's API to analyze the image file and extract the text data from the image. For example, when using Tesseract, execute a command like "tesseract image.png output -l eng --psm 1". The input for this step is the image file, and the output is digital text data.

[0691] Step 5: Perform layout analysis

[0692] The server performs layout analysis based on the positional information of the character data extracted by OCR. Specifically, it analyzes the coordinate information of each text field and executes an algorithm to identify the document layout. The input to this step is the character data and its positional information, and the output is a digital input form based on the layout information.

[0693] Step 6: Generate a digital input form

[0694] The server generates a digital input form based on the layout analysis results. Specifically, it dynamically generates a form template and embeds the analysis results. The input for this step is layout information and character data, and the output is the HTML or JSON data of the digital input form.

[0695] Step 7: Submitting the digital input form

[0696] The server sends the generated digital input form to the user's device. Specifically, it sends data encoded in HTML or JSON format as an HTTP response. The input of this step is the data of the digital input form, and the output is the digital input form displayed on the user's device.

[0697] Step 8: Enter your data

[0698] The user enters data into a digital input form displayed on the terminal display. Specifically, data is entered into form elements such as text boxes and drop-down lists. The input of this step is the user's input data, and the output is the data entered into the form.

[0699] Step 9: Sending Data

[0700] After the user has completed entering data, they click the "Submit" button to send the input data to the server. Specifically, the form data is sent to the server as an HTTP POST request. The input of this step is the data entered by the user, and the output is the data sent to the server.

[0701] Step 10: Save your data

[0702] The server stores the received data in a database. Specifically, it uses a relational database management system (e.g., MySQL or PostgreSQL) to add records to the database by executing SQL INSERT statements. The input to this step is the data sent to the server, and the output is the records stored in the database.

[0703] (Application example 1)

[0704] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0705] When digitizing paper documents, manual data entry is extremely time-consuming and labor-intensive, and prone to human error. In addition, factories need to manage a wide range of paper documents, making efficient digitization a necessity. Furthermore, mechanical methods for converting paper data into digital format are not being used effectively.

[0706] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0707] In this invention, the server includes means for a user to photograph an image of a paper document and upload the image to the system, means for the server to receive the image and extract text data from the image using optical character recognition means, means for the server to analyze the extracted text data and the layout of the image and automatically generate a digital input form, means for sending the generated digital input form to the user's terminal and displaying it on the terminal, means for sending data entered by the user on the terminal to the server and storing it in a database, means for a robot to photograph a paper document and upload the image to the server, means for the robot to display the generated digital input form and for an operator to enter data, and means for sending the entered data to the server and storing it in a database, thereby enabling efficient and accurate digitization of paper documents.

[0708] "User" means a person or entity that uses the System to digitize paper documents.

[0709] "Paper documents" refers to documents or documents printed or recorded on paper, such as invoices, receipts, manuals, and checklists.

[0710] An "image" is digital visual information captured by a camera of a paper document.

[0711] In the present invention, the term "system" refers to a set of processes and devices for digitizing paper documents.

[0712] A "server" is a computing resource that analyzes received images, generates digital input forms, and stores data.

[0713] "Optical character recognition" is a technology for analyzing text within an image and converting it into digital character data.

[0714] A "digital input form" is a digital form into which a user can input data, constructed based on extracted text data and layout analysis.

[0715] A "robot" is a mechanical device that automatically photographs paper documents in factories and other work sites and uploads the images to the system.

[0716] A "database" is a system that stores input data and allows it to be managed and searched as needed.

[0717] A "worker" is a person whose role is to operate a robot or enter data into a digital input form.

[0718] MODE FOR CARRYING OUT THE INVENTION

[0719] This invention is a system that digitizes paper documents and enables efficient and accurate information collection. The system starts when a user takes a photo of a paper document and uploads the image to a server using a robot or terminal.

[0720] The server first receives the image data and extracts the text data from the image using optical character recognition (OCR). Specifically, an OCR library such as Tesseract is used. The extracted text data and the image layout are then analyzed to generate a digital input form. OpenCV is used to preprocess the image, removing noise and detecting contours, enabling accurate text extraction.

[0721] The generated digital input form is sent to the user's terminal or robot and displayed. The user or worker enters data on these terminals or devices, and the data is then sent back to the server and stored in a database. This makes data entry from paper more efficient and reduces input errors.

[0722] This system is particularly effective in factories. The process of factory robots automatically photographing and digitizing paper checklists dramatically improves on-site data management. Furthermore, by analyzing the layout of the input form, a digital form is created that closely matches the format of the original paper document, allowing workers to enter data without any discomfort.

[0723] A concrete example would be the use of a checklist in a factory. For example, if there is a checklist like this:

[0724] Machine name: XYZ product

[0725] Check item: Oil level

[0726] Inspection result: Pass Fail

[0727] The robot takes a photo of this checklist and uploads it to the system. The text is analyzed by Tesseract and sent to the server, where an input form is generated based on the layout analysis.

[0728] Examples of prompts include:

[0729] Generate a digital input form from the following text:

[0730] Machine name: XYZ product

[0731] Check item: Oil level

[0732] Inspection result: Pass Fail

[0733] The generated form should have the following format:

[0734] {

[0735] "formFields": [

[0736] {"label": "machine name", "type": "text"},

[0737] {"label": "Inspection items", "type": "text"},

[0738] {"label": "Inspection result", "type": "radio", "options": ["Pass", "Fail"]}

[0739] ]

[0740] }

[0741] Using these prompts, accurate digital input forms are automatically generated. The data is then stored in a database for further processing and analysis. This system allows for the rapid and accurate digitization of paper-based data.

[0742] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0743] Step 1:

[0744] A user takes an image of a paper document.

[0745] Input: Paper documents

[0746] Output: Captured image data (e.g., JPEG image file)

[0747] Specific operation: The user uses the camera on their smartphone or tablet to take a picture of the checklist or manual.

[0748] Step 2:

[0749] The robot uploads the captured images to the system.

[0750] Input: Photographed image data

[0751] Output: Image data sent to the server

[0752] Specific operation: Images taken by the camera mounted on the robot are transferred to the built-in control unit and uploaded to a server via the network.

[0753] Step 3:

[0754] The server receives the uploaded images and temporarily stores them in a database.

[0755] Input: Uploaded image data

[0756] Output: Temporarily saved image data

[0757] Specific operation: The server receives the HTTP request and saves the image data in a designated folder for temporary storage.

[0758] Step 4:

[0759] The server performs OCR (optical character recognition) and extracts text data from the image data.

[0760] Input: Temporarily saved image data

[0761] Output: Extracted text data

[0762] How it works: The server uses an OCR library such as Tesseract to analyze the letters and numbers in the image and convert them into digital text.

[0763] Step 5:

[0764] The server performs layout analysis and generates a digital input form.

[0765] Input: Extracted text data and image layout information

[0766] Output: Generated digital input form (e.g. JSON format)

[0767] Specific operation: The server uses image processing libraries such as OpenCV to analyze the positional information of the text and generate a digital input form that closely resembles the layout of paper media.

[0768] Step 6:

[0769] The server sends the generated digital input form to the terminal or robot.

[0770] Input: Generated digital input form

[0771] Output: Submitted digital input form

[0772] Specific operation: The server sends the generated form to the terminal or robot as an HTTP response.

[0773] Step 7:

[0774] Users or workers enter data into a digital input form displayed on a terminal or robot display.

[0775] Input: Digital input form

[0776] Output: The data entered (e.g., the value of a form field)

[0777] Specific operation: The user or worker uses the touch panel or keyboard of the terminal or robot to enter the required data into the displayed form.

[0778] Step 8:

[0779] The terminal or robot sends the entered data to the server.

[0780] Input: Data entered

[0781] Output: Data sent to the server

[0782] Specific operation: When a user or worker presses the "Send" button, the entered data is sent to the server as an HTTP request.

[0783] Step 9:

[0784] The server stores the received data in a database.

[0785] Input: Data sent to the server

[0786] Output: Data stored in the database

[0787] What happens: The server converts the received data into an appropriate format and inserts it into a database, where it can be used for later analysis, reporting, etc.

[0788] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0789] The present invention is a system that digitizes paper documents, eliminates the need for manual data entry, and enables efficient and accurate information collection, and further includes a means for recognizing the user's emotions and reflecting the results in the system. Specific embodiments of the various means are described below.

[0790] Program processing overview

[0791] Image upload

[0792] 1. The user takes a picture on the device

[0793] Users use the camera on their smartphone or tablet to take a picture of a paper document (such as an invoice or receipt) using either a dedicated camera app or the device's built-in camera app.

[0794] 2. User uploads image to the system

[0795] The user launches a dedicated application, selects the captured image, and presses the "upload" button to send the image to the system.

[0796] Image processing, analysis and emotion recognition

[0797] 3. The server receives the image

[0798] The server receives the image sent from the user's device, temporarily stores it in storage, and returns a response confirming receipt to the device.

[0799] 4. The server runs the OCR engine

[0800] The server runs an optical character recognition (OCR) engine on the received image to extract text data from the image. For example, if the server uses the Tesseract OCR engine, it calls its API to process the data.

[0801] 5. The server performs layout analysis

[0802] The server analyzes the layout of the document based on the location information of the text data, and as a result, the digital input form is reproduced with the same layout as the paper document.

[0803] 6. The server runs the emotion engine

[0804] The server analyzes the user's image and input actions and activates an emotion engine that recognizes the user's emotions. For example, it uses an emotion recognition API to analyze emotions from the user's facial expressions and input speed.

[0805] Input form generation and display

[0806] 7. The server generates the digital input form

[0807] The server generates a digital input form based on the analysis results, which has the same layout (position of input fields, item names, etc.) as the original paper document.

[0808] 8. The server encodes the input form

[0809] The server encodes the generated digital input form in a format such as JSON or HTML and prepares it to be sent to the user's device.

[0810] 9. The server sends the input form to the device

[0811] The server transmits the encoded digital input form data to the user's terminal, including the transmitted data as a response.

[0812] 10. The device receives the input form

[0813] The terminal receives and decodes the digital input form data sent from the server, thereby preparing to display the digital input form on the terminal.

[0814] 11. The device displays the input form

[0815] The terminal displays the received digital input form on the screen so that the user can fill it in. The user interface is designed to be intuitive.

[0816] Data entry and transmission and emotional feedback

[0817] 12. User enters data into input form

[0818] The user inputs necessary data into the digital input form displayed on the terminal, such as product name, quantity, and price.

[0819] 13. Feedback from the Emotion Engine

[0820] The emotion engine dynamically changes the design and message as you type, reflecting your emotions. For example, if you're feeling stressed, it will present a relaxing message or color tone.

[0821] 14. Save or submit user-entered data

[0822] After completing the data entry, the user presses the "Save" or "Send" button. If the "Save" button is pressed, the data is temporarily saved on the terminal, and if the "Send" button is pressed, the data is sent to the server.

[0823] 15. The device sends the input data to the server

[0824] When the terminal transmits the input data to the server via an HTTP request, the terminal encrypts the data as necessary.

[0825] 16. The server receives and stores the data

[0826] The server validates the data received from the devices and stores it in a database for further processing and report generation.

[0827] 17. The server performs data analysis and report generation

[0828] The server performs statistical processing and generates reports based on the stored data, such as aggregating sales data and generating monthly reports.

[0829] Specific examples

[0830] Retailer Use

[0831] For example, this system would be extremely useful for retailers to easily digitize their daily sales data. Retailers' staff members take photos of each customer's receipts with their smartphones and upload them. The server analyzes the images and generates a digital receipt entry form. The staff member enters the data on the device and stores it on the server, making it easy to digitize sales data. In addition, the emotion engine detects the staff member's stress and provides feedback to help them relax.

[0832] Use of medical facilities

[0833] In medical facilities, patient medical records are often kept on paper. By using this system, medical records are digitized and managed efficiently. Medical staff take photos of paper medical records and upload them to the system. The server analyzes the images and generates a digital medical record entry form. Staff enter the necessary data into the form and save it on the server, further increasing the digitization of medical records. An emotion engine detects staff fatigue and provides refreshing messages, improving work efficiency and comfort.

[0834] These processes enable rapid and accurate digitization of paper documents, improving work efficiency and ensuring accurate data management. Furthermore, the use of an emotion engine is expected to reduce user stress and improve work efficiency.

[0835] The processing flow will be explained below.

[0836] Step 1:

[0837] The user takes a picture on the device

[0838] Users use the camera on their smartphone or tablet to take a picture of a paper document (such as an invoice or receipt) using either a dedicated camera app or the device's built-in camera app.

[0839] Step 2:

[0840] User uploads an image to the system

[0841] The user launches a dedicated application, selects the captured image, and presses the "upload" button to send the image to the system.

[0842] Step 3:

[0843] The device sends the image to the server

[0844] The device uses an HTTP request (such as a POST request) to send the captured image to the server, compressing or encrypting the image data as necessary.

[0845] Step 4:

[0846] The server receives the image

[0847] The server receives the image sent by the user, temporarily stores it in storage, and returns a response confirming receipt to the terminal.

[0848] Step 5:

[0849] The server runs the OCR engine

[0850] The server runs an optical character recognition (OCR) engine on the received image to extract text data from the image. For example, if the server uses the Tesseract OCR engine, it calls its API to process the data.

[0851] Step 6:

[0852] The server performs the layout analysis

[0853] The server analyzes the layout of the document based on the location information of the text data, and as a result, the digital input form is reproduced with the same layout as the paper document.

[0854] Step 7:

[0855] The server runs the emotion engine

[0856] The server analyzes the user's image and input actions and activates an emotion engine that recognizes the user's emotions. For example, it uses an emotion recognition API to analyze emotions from the user's facial expressions and input speed.

[0857] Step 8:

[0858] The server generates the digital input form

[0859] The server generates a digital input form based on the analysis results, which has the same layout (position of input fields, item names, etc.) as the original paper document.

[0860] Step 9:

[0861] The server encodes the input form

[0862] The server encodes the generated digital input form in a format such as JSON or HTML and prepares it to be sent to the user's device.

[0863] Step 10:

[0864] The server sends the input form to the terminal

[0865] The server transmits the encoded digital input form data to the user's terminal, including the transmitted data as a response.

[0866] Step 11:

[0867] The device receives the input form

[0868] The terminal receives and decodes the digital input form data sent from the server, thereby preparing to display the digital input form on the terminal.

[0869] Step 12:

[0870] The device displays the input form

[0871] The terminal displays the received digital input form on the screen so that the user can fill it in. The user interface is designed to be intuitive.

[0872] Step 13:

[0873] The user enters data into the input form

[0874] The user inputs necessary data into the digital input form displayed on the terminal, such as product name, quantity, and price.

[0875] Step 14:

[0876] Feedback from the Emotion Engine

[0877] The server's emotion engine monitors the user's facial expressions and behavior while they are typing and evaluates them in real time. For example, if the user is feeling stressed, a relaxing message or color tone will be dynamically displayed on the input form.

[0878] Step 15:

[0879] User saves or submits input data

[0880] After completing the data entry, the user presses the "Save" or "Send" button. When the "Save" button is pressed, the data is temporarily saved on the terminal, and when the "Send" button is pressed, the data is sent to the server.

[0881] Step 16:

[0882] The device sends the input data to the server

[0883] The terminal sends the input data to the server via an HTTP request, encrypting the data if necessary.

[0884] Step 17:

[0885] The server receives and stores the data

[0886] The server verifies the data received from the devices and stores it in a database for further processing and report generation.

[0887] Step 18:

[0888] The server performs data analysis and report generation

[0889] The server performs statistical processing and generates reports based on the stored data, such as aggregating sales data and generating monthly reports.

[0890] Through these steps, paper documents can be digitized quickly and accurately, improving work efficiency and data management. Furthermore, the use of an emotion engine is expected to reduce user stress and improve work efficiency.

[0891] Example 2

[0892] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0893] Digitizing paper documents requires manual data entry, which is time-consuming and labor-intensive. Furthermore, traditional systems often have difficulty accurately reproducing the layout of documents. Furthermore, there is no feedback based on the user's emotions or state, resulting in a lack of consideration for user stress and work efficiency.

[0894] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for extracting text data from an image using optical character recognition means, means for automatically generating a digital input form by analyzing the extracted text data and the layout of the image, and means for analyzing the user's image and input actions, recognizing the user's emotions, and providing feedback. This makes it possible to efficiently and accurately digitize paper documents, further reducing user stress and improving work efficiency.

[0895] "User" means any person or legal entity that uses the System to digitize paper documents.

[0896] A "paper document" is a document containing information printed or handwritten on paper.

[0897] "Means for taking images" refers to a method or device for taking images of paper documents using a device equipped with a camera function (such as a smartphone or tablet).

[0898] "System" means a set of devices and programs, including servers and terminals, for digitizing paper documents and processing and storing data.

[0899] A "server" is a computer that receives images, processes data, and stores it.

[0900] "Optical character recognition" refers to technology or equipment that analyzes character information in an image and extracts it as text data. An example of this is an OCR (optical character recognition) engine.

[0901] "Text data" refers to character information extracted by optical character recognition means.

[0902] A "layout analysis means" is a technique or device that analyzes the structure and layout of a document based on the positional information of text data within an image.

[0903] A "digital input form" is an input form that is reproduced in a digital environment based on a paper document.

[0904] "Means for providing feedback" refers to technologies or devices that provide appropriate responses or advice in real time based on the user's emotions and input actions.

[0905] "Encryption means" refers to a technique or device that encrypts data to ensure data security.

[0906] A "database" is a storage device or system for storing and managing data received by a server.

[0907] "Emotion recognition means" refers to technology or devices that analyze and evaluate a user's emotional state based on the user's image and input actions.

[0908] This system aims to digitize paper documents, improve work efficiency, and organize data accurately. Furthermore, it analyzes user emotions and provides feedback to improve usability. This system operates in cooperation with users, terminals, and servers.

[0909] Hardware and software used

[0910] Hardware: Smartphones, tablets, and server computers used by users

[0911] Software: Camera app (dedicated app or device-specific app), OCR engine (e.g., Tesseract), emotion recognition API, database system

[0912] Program processing overview

[0913] 1. The user takes a picture on the device

[0914] The user uses the camera on their smartphone or tablet to take a picture of a paper document (e.g., an invoice or receipt) using a dedicated app or the device's standard camera app.

[0915] 2. User uploads image to the system

[0916] The user launches a dedicated application, selects the captured image, and presses the "upload" button to send the image to the system.

[0917] 3. The server receives the image

[0918] The server receives the image sent from the user's device, temporarily stores it in storage, and returns a response confirming receipt to the device.

[0919] 4. The server runs the OCR engine

[0920] The server uses an optical character recognition (OCR) engine to extract text data from the image. Specifically, it calls the API of the Tesseract OCR engine to obtain the character information in the image as text data.

[0921] 5. The server performs layout analysis

[0922] The server analyzes the layout based on the location information of the extracted text data and generates a digital input form, which creates a digital form that retains the same layout as the paper document.

[0923] 6. The server runs the emotion engine

[0924] The server analyzes the user's image and input actions to activate an emotion engine that recognizes emotions. Specifically, it uses an emotion recognition API to analyze the user's emotions based on their facial expressions and input speed, and provides feedback.

[0925] 7. The server generates and submits the digital input form

[0926] The server generates a digital input form based on the analysis results, encodes this form in JSON or HTML format, and sends it to the user's device.

[0927] 8. The device receives and displays the input form

[0928] The terminal receives the digital input form data sent from the server, decodes it, and displays it, allowing the user to enter the required data into the digital input form.

[0929] 9. The device sends the input data to the server

[0930] After the user has completed the data entry, the data is sent from the terminal to the server, where it is encrypted if necessary.

[0931] 10. The server receives and stores the data

[0932] The server validates the received data and stores it securely in a database for later statistical processing and report generation.

[0933] 11. The server performs data analysis and report generation

[0934] The server performs statistical processing and generates reports based on the stored data, such as compiling sales data and generating monthly reports.

[0935] Specific examples

[0936] For retailers

[0937] This system is extremely useful for retailers digitizing their daily sales data. When a salesperson takes a photo of a customer's receipt with their smartphone and uploads it, the server analyzes the image and generates a digital receipt entry form. The salesperson enters the data on the device and sends it to the server. The system uses an emotion engine to detect the salesperson's stress and provides a relaxing message.

[0938] For medical facilities

[0939] Medical facilities use this system to digitize paper medical records. When medical staff take a photo of a paper medical record and upload it to the system, the server analyzes the image and generates a digital input form. The staff enters data into the form and sends it to the server. An emotion engine detects staff fatigue and displays a refresh message. This allows for efficient management of medical records and improves work comfort.

[0940] In this way, the system of the present invention not only efficiently and accurately digitizes paper documents, but also provides technical means for reducing user stress and improving work efficiency.

[0941] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0942] Step 1:

[0943] The user takes a photo of a paper document

[0944] Input: Paper documents (e.g. invoices, receipts)

[0945] How it works: The user uses the built-in camera on their smartphone or tablet to take a photo of a paper document using a dedicated app or the device's standard camera app.

[0946] Output: Captured image data

[0947] Step 2:

[0948] A user uploads an image to the system

[0949] Input: Captured image data

[0950] How it works: The user launches the dedicated application, selects the captured image, and presses the "Upload" button, which sends the image data to the server via the Internet.

[0951] Output: Image data sent to the server

[0952] Step 3:

[0953] The server receives the image

[0954] Input: Image data sent by the user

[0955] Operation: The server temporarily stores the received image data in storage and returns a response confirming receipt to the user device.

[0956] Output: Image data stored in storage, receipt confirmation response

[0957] Step 4:

[0958] The server runs the OCR engine

[0959] Input: Received and stored image data

[0960] How it works: The server uses an optical character recognition (OCR) engine (e.g., Tesseract) to extract character information from the image data as text data.

[0961] Output: Extracted text data

[0962] Step 5:

[0963] The server performs the layout analysis

[0964] Input: Extracted text and image data

[0965] How it works: The server analyzes the location information of the text data and the layout of the images, and automatically generates a digital input form based on the layout of the paper medium.

[0966] Output: The generated digital input form

[0967] Step 6:

[0968] The server runs the emotion engine

[0969] Input: User images, input data, and action data

[0970] How it works: The server uses the emotion recognition API to analyze the user's facial expressions and input speed to recognize their emotions. Based on the analysis results, it generates feedback content for the user.

[0971] Output: User sentiment analysis results, feedback content

[0972] Step 7:

[0973] The server generates and submits the digital input form

[0974] Input: Digital input form, feedback content

[0975] How it works: The server encodes the generated digital input form in JSON or HTML format and sends it to the user's device, optionally incorporating feedback into the submitted data.

[0976] Output: Encoded digital input form data, feedback data

[0977] Step 8:

[0978] The device receives and displays the input form.

[0979] Input: Digital input form data sent from the server

[0980] Operation: The user terminal receives the image data, decodes it, and displays an input form. The user can then enter the required data into the form.

[0981] Output: Digital input form displayed on the terminal

[0982] Step 9:

[0983] The user enters data into the form

[0984] Input: Digital input forms, data entered by users

[0985] How it works: The user enters data such as product name, quantity, and price into a digital input form displayed on the terminal.

[0986] Output: Data entered by the user

[0987] Step 10:

[0988] The device sends the input data to the server

[0989] Input: Data entered by the user

[0990] How it works: When a user presses the "Save" or "Submit" button, the device sends the input data to the server via an HTTP request, encrypting the data if necessary.

[0991] Output: Input data sent to the server

[0992] Step 11:

[0993] The server receives and stores the data

[0994] Input: Input data sent from the terminal

[0995] Operation: The server stores the received data in a database for later statistical processing and report generation.

[0996] Output: Data stored in the database

[0997] Step 12:

[0998] The server performs data analysis and report generation.

[0999] Input: Data stored in the database

[1000] How it works: The server processes the stored data and generates statistics and reports, such as aggregating sales data and generating monthly reports.

[1001] Output: easy-to-read statistics and reports

[1002] (Application example 2)

[1003] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1004] In food delivery operations, digitizing handwritten order forms and delivery confirmations has become a problem, resulting in time, mistakes, and stress for delivery workers. Furthermore, managing paper documents is complicated, creating a need for efficient data management. Furthermore, there is a lack of methods to reduce the workload and manage stress for delivery workers.

[1005] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for a user to take an image of a paper document and upload the image to the system; means for the server to receive the image and extract text data from the image using optical character recognition means; means for the server to analyze the extracted text data and the image layout and automatically generate a digital input form; means for the server to send the generated digital input form to the user's terminal and display it on the terminal; means for the server to send data entered by the user on the terminal and store it in a database; and means for the server to recognize the user's emotions using image analysis means and provide feedback based on the emotions to the terminal. This allows delivery personnel to efficiently and accurately digitize paper documents and manage stress in real time.

[1006] Definitions of important words

[1007] "User" means a person or organization that uses a system to digitize paper documents.

[1008] A "paper document" is a document containing information printed or handwritten on paper.

[1009] An "image" is visual data captured using a device such as a camera.

[1010] "Upload" is the act of sending data from a user's device to a server.

[1011] "System" refers to a set of technological means, including computers and software.

[1012] A "server" is a computer system that centrally receives, processes, and stores data.

[1013] "Optical character recognition" is a technology that analyzes text within an image and converts it into digital data.

[1014] "Text data" is digital information expressed as a string of characters.

[1015] "Layout" refers to information about the arrangement of text, images, etc. in a paper document.

[1016] "Digital input form" means an electronic form for entering data from a digitized document.

[1017] A "terminal" is an electronic device operated by a user, such as a smartphone or tablet.

[1018] "Image analysis means" is a technique for extracting specific information from an image.

[1019] "Emotion recognition" is a technology that analyzes emotions from a user's facial expressions and behavior.

[1020] "Feedback" refers to the information or response that a system provides to a user.

[1021] A "database" is a system for efficiently storing and managing digital data.

[1022] Description (Mode for carrying out the invention)

[1023] The present invention provides a system that allows users to digitize paper documents and manage information efficiently and accurately. Specific embodiments of the present invention will be described below.

[1024] Hardware and Software Configuration

[1025] Hardware:

[1026] Device: An electronic device such as a smartphone or tablet.

[1027] Server: A computer system that receives, processes, and stores data.

[1028] software:

[1029] Flask: Implementing server programs using a web framework.

[1030] PaddleOCR: An optical character recognition engine for extracting text from images.

[1031] FER: Analyzes emotions from facial expressions in images through an emotion recognition engine.

[1032] OpenCV: Use the image processing library to analyze and process images.

[1033] System Operation

[1034] The server performs the following process.

[1035] 1. Receiving and saving images: This receives images of paper documents taken by users on their devices and temporarily saves them in storage. This is implemented using the Flask framework.

[1036] 2. Text extraction using OCR: PaddleOCR is used to extract text data from images. The OCR engine processes the received image and returns the extracted text data to the server.

[1037] 3. Layout analysis: The text data extracted by OCR is analyzed to recreate the same layout as the original paper document. This processing is performed on the server side, allowing the generation of a digital input form.

[1038] 4. Emotion Recognition: FER is used to analyze the user's emotions. Based on facial expressions in the image and input speed, emotions are analyzed and feedback corresponding to the emotion is provided to the device.

[1039] 5. Sending and displaying the digital input form: After the analysis is complete, the generated digital input form is encoded in JSON or HTML format and sent to the user's device, where it displays the received data and allows the user to enter information.

[1040] Specific examples

[1041] For example, this system is extremely useful when food delivery workers need to digitize handwritten order forms. The delivery worker takes a photo of the order form with their smartphone and uploads it to a server using a dedicated application. The server receives the image, extracts text data using OCR, and generates a digital input form by analyzing the layout. The generated digital input form is then sent to a terminal, where the delivery worker can enter the necessary information. In addition, an emotion recognition engine can provide relaxing messages or feedback if the delivery worker's stress level is high.

[1042] Example prompt sentence:

[1043] "Please recognize characters in an image and convert them into digital text. Also, please analyze the emotions of faces in the image and return the results in JSON format."

[1044] In this way, paper documents can be digitized quickly and accurately, reducing the workload of delivery personnel and improving work efficiency.

[1045] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1046] Program processing steps

[1047] Step 1:

[1048] The user (delivery person) takes a picture of the paper order form using the camera on their smartphone. The image is temporarily saved in the device's storage.

[1049] Input: Paper purchase order

[1050] Output: Image file saved on device

[1051] Step 2:

[1052] The user launches the dedicated application, selects the image they have taken, and presses the "upload" button, which sends the image to the server.

[1053] Input: Image file saved on your smartphone

[1054] Output: Image data sent to the server

[1055] Step 3:

[1056] The server receives the image, temporarily stores it in storage, and returns a reception confirmation response to the device.

[1057] Input: Image data sent by the user

[1058] Output: Image file saved on the server and a receipt confirmation response to the device

[1059] Step 4:

[1060] The server runs an OCR engine (PaddleOCR) to extract text data from the received images.

[1061] Input: Image file stored on the server

[1062] Output: Extracted text data

[1063] Step 5:

[1064] The server analyzes the layout and generates a digital input form based on the positional information of the text data, recreating the same layout (position of input fields, item names, etc.) as the original paper document.

[1065] Input: Text data extracted by OCR

[1066] Output: Digital input form data

[1067] Step 6:

[1068] The server runs an emotion recognition engine (FER) to analyze the user's emotions from their facial expressions and input actions, and generates appropriate feedback and sends it to the user's device.

[1069] Input: Received image and user input action data

[1070] Output: User sentiment analysis results and feedback

[1071] Step 7:

[1072] The server encodes the generated digital input form in JSON or HTML format and sends it to the user's device. The device then decodes the received digital input form data and displays it on the screen.

[1073] Input: Digital input form data

[1074] Output: A digital input form displayed on the user's device

[1075] Step 8:

[1076] The user enters the necessary data into the digital input form displayed on the terminal, such as product information, quantity, and price.

[1077] Input: User-entered data

[1078] Output: Input digital data

[1079] Step 9:

[1080] The user presses a button to save or send the input data. If the save button is pressed, the data is temporarily saved on the terminal, and if the send button is pressed, the data is sent to the server.

[1081] Input: User input data and button operations

[1082] Output: Data stored on the device or sent to a server

[1083] Step 10:

[1084] The server stores the received data in a database and performs statistical processing and generates reports as needed.

[1085] Input: Transmitted digital data

[1086] Output: Data stored in the database and generated reports

[1087] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1088] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1089] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1090] [Third embodiment]

[1091] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1092] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1093] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1094] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1095] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1096] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1097] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1098] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1099] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1100] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1101] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1102] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1103] The present invention is a system that digitizes paper documents, eliminates the need for manual data entry, and enables efficient and accurate information collection. This system functions as follows.

[1104] Program processing overview

[1105] Image upload

[1106] 1. The user takes a picture on the device

[1107] Users use the camera on their smartphone or tablet to take a picture of a paper document, such as an invoice or receipt.

[1108] 2. User uploads image to the system

[1109] Users upload the images they have taken to the system through a dedicated application. At this time, users press the "upload" button to send the image files to the server.

[1110] Image processing and analysis

[1111] 3. The server receives the image

[1112] The server receives the image sent from the user's device and temporarily stores it for further processing.

[1113] 4. The server performs optical character recognition (OCR)

[1114] The server runs an OCR engine on the received image, analyzes the text information in the image, and extracts the text data in a digital format, possibly using an OCR library such as Tesseract.

[1115] 5. The server performs layout analysis

[1116] The server analyzes the layout based on the location information of the text data and identifies the structure of the input form, allowing the digital input form to be reproduced with the same layout as the paper document.

[1117] Input form generation

[1118] 6. The server generates the digital input form

[1119] The server generates a template based on the analysis results and creates an input form with a layout similar to the original document.

[1120] 7. The server sends the input form to the device

[1121] The server sends the generated digital input form to the user's device, where it is encoded in JSON or HTML and displayed on the user's device.

[1122] Data entry and saving

[1123] 8. User enters data on the device

[1124] The user enters data into a digital input form displayed on the terminal, such as invoice item names, quantities, and amounts.

[1125] 9. User submits input data

[1126] After completing the data entry, the user presses the "Submit" button to send the entered data to the server.

[1127] 10. The server stores the data

[1128] The server stores the received data in a database and manages the necessary information for further processing and report generation.

[1129] Specific examples

[1130] Retailer Use

[1131] For example, this system is very useful for retailers to easily digitize their daily sales data. Retailers' staff take photos of each customer's receipt with their smartphones and upload them. The server analyzes the images and generates a digital receipt entry form. The staff enters the data on the device and saves it on the server, making it easy to digitize sales data.

[1132] Use of medical facilities

[1133] In medical facilities, patient medical records are often kept on paper. By using this system, medical records are digitized and managed efficiently. Medical staff take photos of paper medical records and upload them to the system. The server analyzes the images and generates a digital medical record entry form. Staff enter the necessary data into the form and save it on the server, furthering the digitization of medical records.

[1134] These processes enable paper documents to be digitized quickly and accurately, resulting in more efficient operations and accurate data management.

[1135] The processing flow will be explained below.

[1136] Step 1:

[1137] The user takes a photo on the device

[1138] Users use the camera on their smartphone or tablet to take a picture of a paper document (such as an invoice or receipt) using either a dedicated camera app or the device's standard camera app.

[1139] Step 2:

[1140] User uploads an image to the system

[1141] The user launches a dedicated application, selects the captured image, and presses the "upload" button to send the image to the system.

[1142] Step 3:

[1143] The device sends the image to the server

[1144] The device uses an HTTP request (such as a POST request) to send the captured image to the server, compressing or encrypting the image data as necessary.

[1145] Step 4:

[1146] The server receives the image

[1147] The server receives the image sent by the user, temporarily stores it in storage, and returns a response confirming receipt to the terminal.

[1148] Step 5:

[1149] The server runs the OCR engine

[1150] The server runs an optical character recognition (OCR) engine on the received image to extract text data from the image. For example, if the server uses the Tesseract OCR engine, it calls its API to process the data.

[1151] Step 6:

[1152] The server performs the layout analysis

[1153] The server analyzes the document layout based on the location information of the extracted text data, in order to identify the structure of tables and the positions of input fields, etc., and obtain data for generating a digital input form.

[1154] Step 7:

[1155] The server generates the digital input form

[1156] The server generates a digital input form based on the analysis results, which has the same layout (position of input fields, item names, etc.) as the original paper document.

[1157] Step 8:

[1158] The server encodes the input form

[1159] The server encodes the generated digital input form in a format such as JSON or HTML and prepares it to be sent to the user's device.

[1160] Step 9:

[1161] The server sends the input form to the terminal

[1162] The server transmits the encoded digital input form data to the user's terminal, including the transmitted data as a response.

[1163] Step 10:

[1164] The device receives the input form

[1165] The terminal receives and decodes the digital input form data sent from the server, thereby preparing to display the digital input form on the terminal.

[1166] Step 11:

[1167] The device displays the input form

[1168] The terminal displays the received digital input form on the screen so that the user can fill it in. The user interface is designed to be intuitive.

[1169] Step 12:

[1170] The user enters data into the input form

[1171] The user inputs necessary data into the digital input form displayed on the terminal, such as product name, quantity, and price.

[1172] Step 13:

[1173] User saves or submits input data

[1174] After completing the data entry, the user presses the "Save" or "Send" button. If the "Save" button is pressed, the data is temporarily saved on the terminal, and if the "Send" button is pressed, the data is sent to the server.

[1175] Step 14:

[1176] The device sends the input data to the server

[1177] The terminal sends the input data to the server via an HTTP request, encrypting the data if necessary.

[1178] Step 15:

[1179] The server receives and stores the data

[1180] The server validates the data received from the devices and stores it in a database for further processing and report generation.

[1181] Step 16:

[1182] The server performs data analysis and report generation

[1183] The server performs statistical processing and generates reports based on the stored data, such as aggregating sales data and generating monthly reports.

[1184] Through these steps, paper documents can be digitized quickly and accurately, resulting in more efficient operations and accurate data management.

[1185] Example 1

[1186] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1187] Conventional information collection and management systems required a huge amount of manual work to digitize paper documents, resulting in a lack of efficiency. Furthermore, manual data collection and management entailed the risk of input errors and data leaks, making it unreliable. Furthermore, there were insufficient methods for accurately digitizing paper documents and managing them electronically.

[1188] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1189] In this invention, the server includes means for a user to take an image of a document with an image input device and upload the image to a data processing device, means for the data processing device to receive the image and extract character data from the image using optical character recognition means, means for the data processing device to analyze the extracted character data and the layout of the image and automatically generate a digital input form, means for transmitting the generated digital input form to a user interface for the user and displaying it on the user interface, and means for transmitting data entered by the user on the user interface to the data processing device and storing it in a database, thereby enabling rapid and accurate digitization of paper documents and efficient information collection and management.

[1190] An "image input device" is a device with a camera function, including a smartphone or tablet terminal.

[1191] "Document" refers to information recorded on paper, including bills, receipts, medical records, etc.

[1192] A "data processing device" is a computer system for processing and analyzing image data and text data.

[1193] "Optical character recognition" means software or hardware technology for converting character information in an image into digital text, such as an OCR engine.

[1194] "Character data" is character information in digital form extracted by optical character recognition means.

[1195] "Layout" refers to the position information of character data within an image, including the layout of a document and the position of fields.

[1196] A "digital input form" is an electronic input screen generated based on analyzed character data and layout information.

[1197] A "user interface" refers to the display elements of a terminal screen or application that a user operates and uses to input data.

[1198] "Database" means a structured computer system for storing and managing processed data, including relational database systems.

[1199] "Encryption" means a technique for securely transforming data for the purpose of protecting the data during transmission.

[1200] This invention is a system that digitizes paper documents, eliminates the need for manual data entry, and enables efficient and accurate information collection. This system uses the following hardware and software:

[1201] This system starts with the user taking a photo of a paper document using an image input device (such as a smartphone or tablet), allowing the user to capture paper documents such as invoices, receipts, and medical records as image data.

[1202] Next, the user uploads the captured image to the data processing device. Using a dedicated application, the user selects an image and taps the upload button. The application then sends the image data to the server via an HTTP POST request.

[1203] The server first receives the image data and temporarily stores it. Then, the server launches an OCR engine (e.g., Tesseract) to extract the text data from the image. Specifically, it calls the OCR engine's API, reads the image file, and converts the text data into a digital format. For example, to use Tesseract OCR, run a command like "tesseract image.png output -l eng --psm 1."

[1204] The server then performs layout analysis based on the positional information of the character data. This analysis identifies the position of each text field and automatically generates a digital input form that is displayed in a format similar to the layout of the original paper document.

[1205] The server sends this digital input form to the user's device, specifically, it is encoded in HTML or JSON format and sent as an HTTP response, and the user's device receives this data and displays it on the application.

[1206] The user enters the required data into the digital input form displayed on the terminal, such as the invoice item name, quantity, amount, etc. After completion, the user presses the "Submit" button to send the data to the server, which then sends the data back to the server via an HTTP POST request.

[1207] The server stores the received data in a database using a relational database management system (e.g., MySQL or PostgreSQL) to store and manage the data accurately and efficiently. For example, it executes an INSERT statement to insert the data into the database.

[1208] Specific examples

[1209] Retailer Use

[1210] For example, this system is very useful for retailers who want to digitize their daily sales data. A retailer's employee takes a photo of each customer's receipt with their smartphone and uploads the image to a server using an application. The server analyzes the image and generates a digital receipt entry form. The employee enters the data on the device and presses the "send" button to send the data to the server. The server stores the received data in a database, making it easy to digitize sales data.

[1211] Use of medical facilities

[1212] In medical facilities, patient medical records are often kept on paper. By using this system, medical records are digitized and managed efficiently. Medical staff take photos of paper medical records and upload them to the system using a dedicated application. The server analyzes the images and generates a digital medical record entry form. Staff enter the necessary data into the form on their device, press the "send" button, and send it to the server, thereby advancing the digitization of medical records.

[1213] Prompt Sentence Examples

[1214] "Please explain the system a retailer uses to digitize sales data. The system consists of image capture, upload, OCR analysis, input form generation, data entry, and storage."

[1215] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1216] Step 1: Take an image

[1217] The user takes an image of a paper document using an image input device (smartphone or tablet). The user launches a camera app, adjusts the camera position so that the entire document fits in the image, and presses the capture button. The input for this step is the paper document, and the output is image data (JPEG or PNG format).

[1218] Step 2: Upload your images to the system

[1219] The user launches the dedicated application, selects the captured image, and taps the upload button. The application sends the selected image data to the server as an HTTP POST request. The input of this step is the image data, and the output is the image data sent to the server.

[1220] Step 3: Receive and save the image

[1221] The server receives the image data sent from the user's device and temporarily stores it. Specifically, the server receives the image data using XMLHttpRequest or Fetch API and stores it in a local temporary directory. The input to this step is the image data as an HTTP POST request, and the output is an image file stored in the server's temporary directory.

[1222] Step 4: Perform Optical Character Recognition (OCR)

[1223] The server invokes an OCR engine (e.g., Tesseract) on the stored image. Specifically, it calls the OCR engine's API to analyze the image file and extract the text data from the image. For example, when using Tesseract, execute a command like "tesseract image.png output -l eng --psm 1". The input for this step is the image file, and the output is digital text data.

[1224] Step 5: Perform layout analysis

[1225] The server performs layout analysis based on the positional information of the character data extracted by OCR. Specifically, it analyzes the coordinate information of each text field and executes an algorithm to identify the document layout. The input to this step is the character data and its positional information, and the output is a digital input form based on the layout information.

[1226] Step 6: Generate a digital input form

[1227] The server generates a digital input form based on the layout analysis results. Specifically, it dynamically generates a form template and embeds the analysis results. The input for this step is layout information and character data, and the output is the HTML or JSON data of the digital input form.

[1228] Step 7: Submitting the digital input form

[1229] The server sends the generated digital input form to the user's device. Specifically, it sends data encoded in HTML or JSON format as an HTTP response. The input of this step is the data of the digital input form, and the output is the digital input form displayed on the user's device.

[1230] Step 8: Enter your data

[1231] The user enters data into a digital input form displayed on the terminal display. Specifically, data is entered into form elements such as text boxes and drop-down lists. The input of this step is the user's input data, and the output is the data entered into the form.

[1232] Step 9: Sending Data

[1233] After the user has completed entering data, they click the "Submit" button to send the input data to the server. Specifically, the form data is sent to the server as an HTTP POST request. The input of this step is the data entered by the user, and the output is the data sent to the server.

[1234] Step 10: Save your data

[1235] The server stores the received data in a database. Specifically, it uses a relational database management system (e.g., MySQL or PostgreSQL) to add records to the database by executing SQL INSERT statements. The input to this step is the data sent to the server, and the output is the records stored in the database.

[1236] (Application example 1)

[1237] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1238] When digitizing paper documents, manual data entry is extremely time-consuming and labor-intensive, and prone to human error. In addition, factories need to manage a wide range of paper documents, making efficient digitization a necessity. Furthermore, mechanical methods for converting paper data into digital format are not being used effectively.

[1239] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1240] In this invention, the server includes means for a user to photograph an image of a paper document and upload the image to the system, means for the server to receive the image and extract text data from the image using optical character recognition means, means for the server to analyze the extracted text data and the layout of the image and automatically generate a digital input form, means for sending the generated digital input form to the user's terminal and displaying it on the terminal, means for sending data entered by the user on the terminal to the server and storing it in a database, means for a robot to photograph a paper document and upload the image to the server, means for the robot to display the generated digital input form and for an operator to enter data, and means for sending the entered data to the server and storing it in a database, thereby enabling efficient and accurate digitization of paper documents.

[1241] "User" means a person or entity that uses the System to digitize paper documents.

[1242] "Paper documents" refers to documents or documents printed or recorded on paper, such as invoices, receipts, manuals, and checklists.

[1243] An "image" is digital visual information captured by a camera of a paper document.

[1244] In the present invention, the term "system" refers to a set of processes and devices for digitizing paper documents.

[1245] A "server" is a computing resource that analyzes received images, generates digital input forms, and stores data.

[1246] "Optical character recognition" is a technology for analyzing text within an image and converting it into digital character data.

[1247] A "digital input form" is a digital form into which a user can input data, constructed based on extracted text data and layout analysis.

[1248] A "robot" is a mechanical device that automatically photographs paper documents in factories and other work sites and uploads the images to the system.

[1249] A "database" is a system that stores input data and allows it to be managed and searched as needed.

[1250] A "worker" is a person whose role is to operate a robot or enter data into a digital input form.

[1251] MODE FOR CARRYING OUT THE INVENTION

[1252] This invention is a system that digitizes paper documents and enables efficient and accurate information collection. The system starts when a user takes a photo of a paper document and uploads the image to a server using a robot or terminal.

[1253] The server first receives the image data and extracts the text data from the image using optical character recognition (OCR). Specifically, an OCR library such as Tesseract is used. The extracted text data and the image layout are then analyzed to generate a digital input form. OpenCV is used to preprocess the image, removing noise and detecting contours, enabling accurate text extraction.

[1254] The generated digital input form is sent to the user's terminal or robot and displayed. The user or worker enters data on these terminals or devices, and the data is then sent back to the server and stored in a database. This makes data entry from paper more efficient and reduces input errors.

[1255] This system is particularly effective in factories. The process of factory robots automatically photographing and digitizing paper checklists dramatically improves on-site data management. Furthermore, by analyzing the layout of the input form, a digital form is created that closely matches the format of the original paper document, allowing workers to enter data without any discomfort.

[1256] A concrete example would be the use of a checklist in a factory. For example, if there is a checklist like this:

[1257] Machine name: XYZ product

[1258] Check item: Oil level

[1259] Inspection result: Pass Fail

[1260] The robot takes a photo of this checklist and uploads it to the system. The text is analyzed by Tesseract and sent to the server, where an input form is generated based on the layout analysis.

[1261] Examples of prompts include:

[1262] Generate a digital input form from the following text:

[1263] Machine name: XYZ product

[1264] Check item: Oil level

[1265] Inspection result: Pass Fail

[1266] The generated form should have the following format:

[1267] {

[1268] "formFields": [

[1269] {"label": "machine name", "type": "text"},

[1270] {"label": "Inspection items", "type": "text"},

[1271] {"label": "Inspection result", "type": "radio", "options": ["Pass", "Fail"]}

[1272] ]

[1273] }

[1274] Using these prompts, accurate digital input forms are automatically generated. The data is then stored in a database for further processing and analysis. This system allows for the rapid and accurate digitization of paper-based data.

[1275] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1276] Step 1:

[1277] A user takes an image of a paper document.

[1278] Input: Paper documents

[1279] Output: Captured image data (e.g., JPEG image file)

[1280] Specific operation: The user uses the camera on their smartphone or tablet to take a picture of the checklist or manual.

[1281] Step 2:

[1282] The robot uploads the captured images to the system.

[1283] Input: Photographed image data

[1284] Output: Image data sent to the server

[1285] Specific operation: Images taken by the camera mounted on the robot are transferred to the built-in control unit and uploaded to a server via the network.

[1286] Step 3:

[1287] The server receives the uploaded images and temporarily stores them in a database.

[1288] Input: Uploaded image data

[1289] Output: Temporarily saved image data

[1290] Specific operation: The server receives the HTTP request and saves the image data in a designated folder for temporary storage.

[1291] Step 4:

[1292] The server performs OCR (optical character recognition) and extracts text data from the image data.

[1293] Input: Temporarily saved image data

[1294] Output: Extracted text data

[1295] How it works: The server uses an OCR library such as Tesseract to analyze the letters and numbers in the image and convert them into digital text.

[1296] Step 5:

[1297] The server performs layout analysis and generates a digital input form.

[1298] Input: Extracted text data and image layout information

[1299] Output: Generated digital input form (e.g. JSON format)

[1300] Specific operation: The server uses image processing libraries such as OpenCV to analyze the positional information of the text and generate a digital input form that closely resembles the layout of paper media.

[1301] Step 6:

[1302] The server sends the generated digital input form to the terminal or robot.

[1303] Input: Generated digital input form

[1304] Output: Submitted digital input form

[1305] Specific operation: The server sends the generated form to the terminal or robot as an HTTP response.

[1306] Step 7:

[1307] Users or workers enter data into a digital input form displayed on a terminal or robot display.

[1308] Input: Digital input form

[1309] Output: The data entered (e.g., the value of a form field)

[1310] Specific operation: The user or worker uses the touch panel or keyboard of the terminal or robot to enter the required data into the displayed form.

[1311] Step 8:

[1312] The terminal or robot sends the entered data to the server.

[1313] Input: Data entered

[1314] Output: Data sent to the server

[1315] Specific operation: When a user or worker presses the "Send" button, the entered data is sent to the server as an HTTP request.

[1316] Step 9:

[1317] The server stores the received data in a database.

[1318] Input: Data sent to the server

[1319] Output: Data stored in the database

[1320] What happens: The server converts the received data into an appropriate format and inserts it into a database, where it can be used for later analysis, reporting, etc.

[1321] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1322] The present invention is a system that digitizes paper documents, eliminates the need for manual data entry, and enables efficient and accurate information collection, and further includes a means for recognizing the user's emotions and reflecting the results in the system. Specific embodiments of the various means are described below.

[1323] Program processing overview

[1324] Image upload

[1325] 1. The user takes a picture on the device

[1326] Users use the camera on their smartphone or tablet to take a picture of a paper document (such as an invoice or receipt) using either a dedicated camera app or the device's built-in camera app.

[1327] 2. User uploads image to the system

[1328] The user launches a dedicated application, selects the captured image, and presses the "upload" button to send the image to the system.

[1329] Image processing, analysis and emotion recognition

[1330] 3. The server receives the image

[1331] The server receives the image sent from the user's device, temporarily stores it in storage, and returns a response confirming receipt to the device.

[1332] 4. The server runs the OCR engine

[1333] The server runs an optical character recognition (OCR) engine on the received image to extract text data from the image. For example, if the server uses the Tesseract OCR engine, it calls its API to process the data.

[1334] 5. The server performs layout analysis

[1335] The server analyzes the layout of the document based on the location information of the text data, and as a result, the digital input form is reproduced with the same layout as the paper document.

[1336] 6. The server runs the emotion engine

[1337] The server analyzes the user's image and input actions and activates an emotion engine that recognizes the user's emotions. For example, it uses an emotion recognition API to analyze emotions from the user's facial expressions and input speed.

[1338] Input form generation and display

[1339] 7. The server generates the digital input form

[1340] The server generates a digital input form based on the analysis results, which has the same layout (position of input fields, item names, etc.) as the original paper document.

[1341] 8. The server encodes the input form

[1342] The server encodes the generated digital input form in a format such as JSON or HTML and prepares it to be sent to the user's device.

[1343] 9. The server sends the input form to the device

[1344] The server transmits the encoded digital input form data to the user's terminal, including the transmitted data as a response.

[1345] 10. The device receives the input form

[1346] The terminal receives and decodes the digital input form data sent from the server, thereby preparing to display the digital input form on the terminal.

[1347] 11. The device displays the input form

[1348] The terminal displays the received digital input form on the screen so that the user can fill it in. The user interface is designed to be intuitive.

[1349] Data entry and transmission and emotional feedback

[1350] 12. User enters data into input form

[1351] The user inputs necessary data into the digital input form displayed on the terminal, such as product name, quantity, and price.

[1352] 13. Feedback from the Emotion Engine

[1353] The emotion engine dynamically changes the design and message as you type, reflecting your emotions. For example, if you're feeling stressed, it will present a relaxing message or color tone.

[1354] 14. Save or submit user-entered data

[1355] After completing the data entry, the user presses the "Save" or "Send" button. If the "Save" button is pressed, the data is temporarily saved on the terminal, and if the "Send" button is pressed, the data is sent to the server.

[1356] 15. The device sends the input data to the server

[1357] When the terminal transmits the input data to the server via an HTTP request, the terminal encrypts the data as necessary.

[1358] 16. The server receives and stores the data

[1359] The server validates the data received from the devices and stores it in a database for further processing and report generation.

[1360] 17. The server performs data analysis and report generation

[1361] The server performs statistical processing and generates reports based on the stored data, such as aggregating sales data and generating monthly reports.

[1362] Specific examples

[1363] Retailer Use

[1364] For example, this system would be extremely useful for retailers to easily digitize their daily sales data. Retailers' staff members take photos of each customer's receipts with their smartphones and upload them. The server analyzes the images and generates a digital receipt entry form. The staff member enters the data on the device and stores it on the server, making it easy to digitize sales data. In addition, the emotion engine detects the staff member's stress and provides feedback to help them relax.

[1365] Use of medical facilities

[1366] In medical facilities, patient medical records are often kept on paper. By using this system, medical records are digitized and managed efficiently. Medical staff take photos of paper medical records and upload them to the system. The server analyzes the images and generates a digital medical record entry form. Staff enter the necessary data into the form and save it on the server, further increasing the digitization of medical records. An emotion engine detects staff fatigue and provides refreshing messages, improving work efficiency and comfort.

[1367] These processes enable rapid and accurate digitization of paper documents, improving work efficiency and ensuring accurate data management. Furthermore, the use of an emotion engine is expected to reduce user stress and improve work efficiency.

[1368] The processing flow will be explained below.

[1369] Step 1:

[1370] The user takes a picture on the device

[1371] Users use the camera on their smartphone or tablet to take a picture of a paper document (such as an invoice or receipt) using either a dedicated camera app or the device's built-in camera app.

[1372] Step 2:

[1373] User uploads an image to the system

[1374] The user launches a dedicated application, selects the captured image, and presses the "upload" button to send the image to the system.

[1375] Step 3:

[1376] The device sends the image to the server

[1377] The device uses an HTTP request (such as a POST request) to send the captured image to the server, compressing or encrypting the image data as necessary.

[1378] Step 4:

[1379] The server receives the image

[1380] The server receives the image sent by the user, temporarily stores it in storage, and returns a response confirming receipt to the terminal.

[1381] Step 5:

[1382] The server runs the OCR engine

[1383] The server runs an optical character recognition (OCR) engine on the received image to extract text data from the image. For example, if the server uses the Tesseract OCR engine, it calls its API to process the data.

[1384] Step 6:

[1385] The server performs the layout analysis

[1386] The server analyzes the layout of the document based on the location information of the text data, and as a result, the digital input form is reproduced with the same layout as the paper document.

[1387] Step 7:

[1388] The server runs the emotion engine

[1389] The server analyzes the user's image and input actions and activates an emotion engine that recognizes the user's emotions. For example, it uses an emotion recognition API to analyze emotions from the user's facial expressions and input speed.

[1390] Step 8:

[1391] The server generates the digital input form

[1392] The server generates a digital input form based on the analysis results, which has the same layout (position of input fields, item names, etc.) as the original paper document.

[1393] Step 9:

[1394] The server encodes the input form

[1395] The server encodes the generated digital input form in a format such as JSON or HTML and prepares it to be sent to the user's device.

[1396] Step 10:

[1397] The server sends the input form to the terminal

[1398] The server transmits the encoded digital input form data to the user's terminal, including the transmitted data as a response.

[1399] Step 11:

[1400] The device receives the input form

[1401] The terminal receives and decodes the digital input form data sent from the server, thereby preparing to display the digital input form on the terminal.

[1402] Step 12:

[1403] The device displays the input form

[1404] The terminal displays the received digital input form on the screen so that the user can fill it in. The user interface is designed to be intuitive.

[1405] Step 13:

[1406] The user enters data into the input form

[1407] The user inputs necessary data into the digital input form displayed on the terminal, such as product name, quantity, and price.

[1408] Step 14:

[1409] Feedback from the Emotion Engine

[1410] The server's emotion engine monitors the user's facial expressions and behavior while they are typing and evaluates them in real time. For example, if the user is feeling stressed, a relaxing message or color tone will be dynamically displayed on the input form.

[1411] Step 15:

[1412] User saves or submits input data

[1413] After completing the data entry, the user presses the "Save" or "Send" button. When the "Save" button is pressed, the data is temporarily saved on the terminal, and when the "Send" button is pressed, the data is sent to the server.

[1414] Step 16:

[1415] The device sends the input data to the server

[1416] The terminal sends the input data to the server via an HTTP request, encrypting the data if necessary.

[1417] Step 17:

[1418] The server receives and stores the data

[1419] The server verifies the data received from the devices and stores it in a database for further processing and report generation.

[1420] Step 18:

[1421] The server performs data analysis and report generation

[1422] The server performs statistical processing and generates reports based on the stored data, such as aggregating sales data and generating monthly reports.

[1423] Through these steps, paper documents can be digitized quickly and accurately, improving work efficiency and data management. Furthermore, the use of an emotion engine is expected to reduce user stress and improve work efficiency.

[1424] Example 2

[1425] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1426] Digitizing paper documents requires manual data entry, which is time-consuming and labor-intensive. Furthermore, traditional systems often have difficulty accurately reproducing the layout of documents. Furthermore, there is no feedback based on the user's emotions or state, resulting in a lack of consideration for user stress and work efficiency.

[1427] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for extracting text data from an image using optical character recognition means, means for automatically generating a digital input form by analyzing the extracted text data and the layout of the image, and means for analyzing the user's image and input actions, recognizing the user's emotions, and providing feedback. This makes it possible to efficiently and accurately digitize paper documents, further reducing user stress and improving work efficiency.

[1428] "User" means any person or legal entity that uses the System to digitize paper documents.

[1429] A "paper document" is a document containing information printed or handwritten on paper.

[1430] "Means for taking images" refers to a method or device for taking images of paper documents using a device equipped with a camera function (such as a smartphone or tablet).

[1431] "System" means a set of devices and programs, including servers and terminals, for digitizing paper documents and processing and storing data.

[1432] A "server" is a computer that receives images, processes data, and stores it.

[1433] "Optical character recognition" refers to technology or equipment that analyzes character information in an image and extracts it as text data. An example of this is an OCR (optical character recognition) engine.

[1434] "Text data" refers to character information extracted by optical character recognition means.

[1435] A "layout analysis means" is a technique or device that analyzes the structure and layout of a document based on the positional information of text data within an image.

[1436] A "digital input form" is an input form that is reproduced in a digital environment based on a paper document.

[1437] "Means for providing feedback" refers to technologies or devices that provide appropriate responses or advice in real time based on the user's emotions and input actions.

[1438] "Encryption means" refers to a technique or device that encrypts data to ensure data security.

[1439] A "database" is a storage device or system for storing and managing data received by a server.

[1440] "Emotion recognition means" refers to technology or devices that analyze and evaluate a user's emotional state based on the user's image and input actions.

[1441] This system aims to digitize paper documents, improve work efficiency, and organize data accurately. Furthermore, it analyzes user emotions and provides feedback to improve usability. This system operates in cooperation with users, terminals, and servers.

[1442] Hardware and software used

[1443] Hardware: Smartphones, tablets, and server computers used by users

[1444] Software: Camera app (dedicated app or device-specific app), OCR engine (e.g., Tesseract), emotion recognition API, database system

[1445] Program processing overview

[1446] 1. The user takes a picture on the device

[1447] The user uses the camera on their smartphone or tablet to take a picture of a paper document (e.g., an invoice or receipt) using a dedicated app or the device's standard camera app.

[1448] 2. User uploads image to the system

[1449] The user launches a dedicated application, selects the captured image, and presses the "upload" button to send the image to the system.

[1450] 3. The server receives the image

[1451] The server receives the image sent from the user's device, temporarily stores it in storage, and returns a response confirming receipt to the device.

[1452] 4. The server runs the OCR engine

[1453] The server uses an optical character recognition (OCR) engine to extract text data from the image. Specifically, it calls the API of the Tesseract OCR engine to obtain the character information in the image as text data.

[1454] 5. The server performs layout analysis

[1455] The server analyzes the layout based on the location information of the extracted text data and generates a digital input form, which creates a digital form that retains the same layout as the paper document.

[1456] 6. The server runs the emotion engine

[1457] The server analyzes the user's image and input actions to activate an emotion engine that recognizes emotions. Specifically, it uses an emotion recognition API to analyze the user's emotions based on their facial expressions and input speed, and provides feedback.

[1458] 7. The server generates and submits the digital input form

[1459] The server generates a digital input form based on the analysis results, encodes this form in JSON or HTML format, and sends it to the user's device.

[1460] 8. The device receives and displays the input form

[1461] The terminal receives the digital input form data sent from the server, decodes it, and displays it, allowing the user to enter the required data into the digital input form.

[1462] 9. The device sends the input data to the server

[1463] After the user has completed the data entry, the data is sent from the terminal to the server, where it is encrypted if necessary.

[1464] 10. The server receives and stores the data

[1465] The server validates the received data and stores it securely in a database for later statistical processing and report generation.

[1466] 11. The server performs data analysis and report generation

[1467] The server performs statistical processing and generates reports based on the stored data, such as compiling sales data and generating monthly reports.

[1468] Specific examples

[1469] For retailers

[1470] This system is extremely useful for retailers digitizing their daily sales data. When a salesperson takes a photo of a customer's receipt with their smartphone and uploads it, the server analyzes the image and generates a digital receipt entry form. The salesperson enters the data on the device and sends it to the server. The system uses an emotion engine to detect the salesperson's stress and provides a relaxing message.

[1471] For medical facilities

[1472] Medical facilities use this system to digitize paper medical records. When medical staff take a photo of a paper medical record and upload it to the system, the server analyzes the image and generates a digital input form. The staff enters data into the form and sends it to the server. An emotion engine detects staff fatigue and displays a refresh message. This allows for efficient management of medical records and improves work comfort.

[1473] In this way, the system of the present invention not only efficiently and accurately digitizes paper documents, but also provides technical means for reducing user stress and improving work efficiency.

[1474] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1475] Step 1:

[1476] The user takes a photo of a paper document

[1477] Input: Paper documents (e.g. invoices, receipts)

[1478] How it works: The user uses the built-in camera on their smartphone or tablet to take a photo of a paper document using a dedicated app or the device's standard camera app.

[1479] Output: Captured image data

[1480] Step 2:

[1481] A user uploads an image to the system

[1482] Input: Captured image data

[1483] How it works: The user launches the dedicated application, selects the captured image, and presses the "Upload" button, which sends the image data to the server via the Internet.

[1484] Output: Image data sent to the server

[1485] Step 3:

[1486] The server receives the image

[1487] Input: Image data sent by the user

[1488] Operation: The server temporarily stores the received image data in storage and returns a response confirming receipt to the user device.

[1489] Output: Image data stored in storage, receipt confirmation response

[1490] Step 4:

[1491] The server runs the OCR engine

[1492] Input: Received and stored image data

[1493] How it works: The server uses an optical character recognition (OCR) engine (e.g., Tesseract) to extract character information from the image data as text data.

[1494] Output: Extracted text data

[1495] Step 5:

[1496] The server performs the layout analysis

[1497] Input: Extracted text and image data

[1498] How it works: The server analyzes the location information of the text data and the layout of the images, and automatically generates a digital input form based on the layout of the paper medium.

[1499] Output: The generated digital input form

[1500] Step 6:

[1501] The server runs the emotion engine

[1502] Input: User images, input data, and action data

[1503] How it works: The server uses the emotion recognition API to analyze the user's facial expressions and input speed to recognize their emotions. Based on the analysis results, it generates feedback content for the user.

[1504] Output: User sentiment analysis results, feedback content

[1505] Step 7:

[1506] The server generates and submits the digital input form

[1507] Input: Digital input form, feedback content

[1508] How it works: The server encodes the generated digital input form in JSON or HTML format and sends it to the user's device, optionally incorporating feedback into the submitted data.

[1509] Output: Encoded digital input form data, feedback data

[1510] Step 8:

[1511] The device receives and displays the input form.

[1512] Input: Digital input form data sent from the server

[1513] Operation: The user terminal receives the image data, decodes it, and displays an input form. The user can then enter the required data into the form.

[1514] Output: Digital input form displayed on the terminal

[1515] Step 9:

[1516] The user enters data into the form

[1517] Input: Digital input forms, data entered by users

[1518] How it works: The user enters data such as product name, quantity, and price into a digital input form displayed on the terminal.

[1519] Output: Data entered by the user

[1520] Step 10:

[1521] The device sends the input data to the server

[1522] Input: Data entered by the user

[1523] How it works: When a user presses the "Save" or "Submit" button, the device sends the input data to the server via an HTTP request, encrypting the data if necessary.

[1524] Output: Input data sent to the server

[1525] Step 11:

[1526] The server receives and stores the data

[1527] Input: Input data sent from the terminal

[1528] Operation: The server stores the received data in a database for later statistical processing and report generation.

[1529] Output: Data stored in the database

[1530] Step 12:

[1531] The server performs data analysis and report generation.

[1532] Input: Data stored in the database

[1533] How it works: The server processes the stored data and generates statistics and reports, such as aggregating sales data and generating monthly reports.

[1534] Output: easy-to-read statistics and reports

[1535] (Application example 2)

[1536] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1537] In food delivery operations, digitizing handwritten order forms and delivery confirmations has become a problem, resulting in time, mistakes, and stress for delivery workers. Furthermore, managing paper documents is complicated, creating a need for efficient data management. Furthermore, there is a lack of methods to reduce the workload and manage stress for delivery workers.

[1538] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for a user to take an image of a paper document and upload the image to the system; means for the server to receive the image and extract text data from the image using optical character recognition means; means for the server to analyze the extracted text data and the image layout and automatically generate a digital input form; means for the server to send the generated digital input form to the user's terminal and display it on the terminal; means for the server to send data entered by the user on the terminal and store it in a database; and means for the server to recognize the user's emotions using image analysis means and provide feedback based on the emotions to the terminal. This allows delivery personnel to efficiently and accurately digitize paper documents and manage stress in real time.

[1539] Definitions of important words

[1540] "User" means a person or organization that uses a system to digitize paper documents.

[1541] A "paper document" is a document containing information printed or handwritten on paper.

[1542] An "image" is visual data captured using a device such as a camera.

[1543] "Upload" is the act of sending data from a user's device to a server.

[1544] "System" refers to a set of technological means, including computers and software.

[1545] A "server" is a computer system that centrally receives, processes, and stores data.

[1546] "Optical character recognition" is a technology that analyzes text within an image and converts it into digital data.

[1547] "Text data" is digital information expressed as a string of characters.

[1548] "Layout" refers to information about the arrangement of text, images, etc. in a paper document.

[1549] "Digital input form" means an electronic form for entering data from a digitized document.

[1550] A "terminal" is an electronic device operated by a user, such as a smartphone or tablet.

[1551] "Image analysis means" is a technique for extracting specific information from an image.

[1552] "Emotion recognition" is a technology that analyzes emotions from a user's facial expressions and behavior.

[1553] "Feedback" refers to the information or response that a system provides to a user.

[1554] A "database" is a system for efficiently storing and managing digital data.

[1555] Description (Mode for carrying out the invention)

[1556] The present invention provides a system that allows users to digitize paper documents and manage information efficiently and accurately. Specific embodiments of the present invention will be described below.

[1557] Hardware and Software Configuration

[1558] Hardware:

[1559] Device: An electronic device such as a smartphone or tablet.

[1560] Server: A computer system that receives, processes, and stores data.

[1561] software:

[1562] Flask: Implementing server programs using a web framework.

[1563] PaddleOCR: An optical character recognition engine for extracting text from images.

[1564] FER: Analyzes emotions from facial expressions in images through an emotion recognition engine.

[1565] OpenCV: Use the image processing library to analyze and process images.

[1566] System Operation

[1567] The server performs the following process.

[1568] 1. Receiving and saving images: This receives images of paper documents taken by users on their devices and temporarily saves them in storage. This is implemented using the Flask framework.

[1569] 2. Text extraction using OCR: PaddleOCR is used to extract text data from images. The OCR engine processes the received image and returns the extracted text data to the server.

[1570] 3. Layout analysis: The text data extracted by OCR is analyzed to recreate the same layout as the original paper document. This processing is performed on the server side, allowing the generation of a digital input form.

[1571] 4. Emotion Recognition: FER is used to analyze the user's emotions. Based on facial expressions in the image and input speed, emotions are analyzed and feedback corresponding to the emotion is provided to the device.

[1572] 5. Sending and displaying the digital input form: After the analysis is complete, the generated digital input form is encoded in JSON or HTML format and sent to the user's device, where it displays the received data and allows the user to enter information.

[1573] Specific examples

[1574] For example, this system is extremely useful when food delivery workers need to digitize handwritten order forms. The delivery worker takes a photo of the order form with their smartphone and uploads it to a server using a dedicated application. The server receives the image, extracts text data using OCR, and generates a digital input form by analyzing the layout. The generated digital input form is then sent to a terminal, where the delivery worker can enter the necessary information. In addition, an emotion recognition engine can provide relaxing messages or feedback if the delivery worker's stress level is high.

[1575] Example prompt sentence:

[1576] "Please recognize characters in an image and convert them into digital text. Also, please analyze the emotions of faces in the image and return the results in JSON format."

[1577] In this way, paper documents can be digitized quickly and accurately, reducing the workload of delivery personnel and improving work efficiency.

[1578] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1579] Program processing steps

[1580] Step 1:

[1581] The user (delivery person) takes a picture of the paper order form using the camera on their smartphone. The image is temporarily saved in the device's storage.

[1582] Input: Paper purchase order

[1583] Output: Image file saved on device

[1584] Step 2:

[1585] The user launches the dedicated application, selects the image they have taken, and presses the "upload" button, which sends the image to the server.

[1586] Input: Image file saved on your smartphone

[1587] Output: Image data sent to the server

[1588] Step 3:

[1589] The server receives the image, temporarily stores it in storage, and returns a reception confirmation response to the device.

[1590] Input: Image data sent by the user

[1591] Output: Image file saved on the server and a receipt confirmation response to the device

[1592] Step 4:

[1593] The server runs an OCR engine (PaddleOCR) to extract text data from the received images.

[1594] Input: Image file stored on the server

[1595] Output: Extracted text data

[1596] Step 5:

[1597] The server analyzes the layout and generates a digital input form based on the positional information of the text data, recreating the same layout (position of input fields, item names, etc.) as the original paper document.

[1598] Input: Text data extracted by OCR

[1599] Output: Digital input form data

[1600] Step 6:

[1601] The server runs an emotion recognition engine (FER) to analyze the user's emotions from their facial expressions and input actions, and generates appropriate feedback and sends it to the user's device.

[1602] Input: Received image and user input action data

[1603] Output: User sentiment analysis results and feedback

[1604] Step 7:

[1605] The server encodes the generated digital input form in JSON or HTML format and sends it to the user's device. The device then decodes the received digital input form data and displays it on the screen.

[1606] Input: Digital input form data

[1607] Output: A digital input form displayed on the user's device

[1608] Step 8:

[1609] The user enters the necessary data into the digital input form displayed on the terminal, such as product information, quantity, and price.

[1610] Input: User-entered data

[1611] Output: Input digital data

[1612] Step 9:

[1613] The user presses a button to save or send the input data. If the save button is pressed, the data is temporarily saved on the terminal, and if the send button is pressed, the data is sent to the server.

[1614] Input: User input data and button operations

[1615] Output: Data stored on the device or sent to a server

[1616] Step 10:

[1617] The server stores the received data in a database and performs statistical processing and generates reports as needed.

[1618] Input: Transmitted digital data

[1619] Output: Data stored in the database and generated reports

[1620] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1621] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1622] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1623] [Fourth embodiment]

[1624] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1625] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1626] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1627] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1628] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1629] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1630] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1631] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1632] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1633] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1634] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1635] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1636] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1637] The present invention is a system that digitizes paper documents, eliminates the need for manual data entry, and enables efficient and accurate information collection. This system functions as follows.

[1638] Program processing overview

[1639] Image upload

[1640] 1. The user takes a picture on the device

[1641] Users use the camera on their smartphone or tablet to take a picture of a paper document, such as an invoice or receipt.

[1642] 2. User uploads image to the system

[1643] Users upload the images they have taken to the system through a dedicated application. At this time, users press the "upload" button to send the image files to the server.

[1644] Image processing and analysis

[1645] 3. The server receives the image

[1646] The server receives the image sent from the user's device and temporarily stores it for further processing.

[1647] 4. The server performs optical character recognition (OCR)

[1648] The server runs an OCR engine on the received image, analyzes the text information in the image, and extracts the text data in a digital format, possibly using an OCR library such as Tesseract.

[1649] 5. The server performs layout analysis

[1650] The server analyzes the layout based on the location information of the text data and identifies the structure of the input form, allowing the digital input form to be reproduced with the same layout as the paper document.

[1651] Input form generation

[1652] 6. The server generates the digital input form

[1653] The server generates a template based on the analysis results and creates an input form with a layout similar to the original document.

[1654] 7. The server sends the input form to the device

[1655] The server sends the generated digital input form to the user's device, where it is encoded in JSON or HTML and displayed on the user's device.

[1656] Data entry and saving

[1657] 8. User enters data on the device

[1658] The user enters data into a digital input form displayed on the terminal, such as invoice item names, quantities, and amounts.

[1659] 9. User submits input data

[1660] After completing the data entry, the user presses the "Submit" button to send the entered data to the server.

[1661] 10. The server stores the data

[1662] The server stores the received data in a database and manages the necessary information for further processing and report generation.

[1663] Specific examples

[1664] Retailer Use

[1665] For example, this system is very useful for retailers to easily digitize their daily sales data. Retailers' staff take photos of each customer's receipt with their smartphones and upload them. The server analyzes the images and generates a digital receipt entry form. The staff enters the data on the device and saves it on the server, making it easy to digitize sales data.

[1666] Use of medical facilities

[1667] In medical facilities, patient medical records are often kept on paper. By using this system, medical records are digitized and managed efficiently. Medical staff take photos of paper medical records and upload them to the system. The server analyzes the images and generates a digital medical record entry form. Staff enter the necessary data into the form and save it on the server, furthering the digitization of medical records.

[1668] These processes enable paper documents to be digitized quickly and accurately, resulting in more efficient operations and accurate data management.

[1669] The processing flow will be explained below.

[1670] Step 1:

[1671] The user takes a photo on the device

[1672] Users use the camera on their smartphone or tablet to take a picture of a paper document (such as an invoice or receipt) using either a dedicated camera app or the device's standard camera app.

[1673] Step 2:

[1674] User uploads an image to the system

[1675] The user launches a dedicated application, selects the captured image, and presses the "upload" button to send the image to the system.

[1676] Step 3:

[1677] The device sends the image to the server

[1678] The device uses an HTTP request (such as a POST request) to send the captured image to the server, compressing or encrypting the image data as necessary.

[1679] Step 4:

[1680] The server receives the image

[1681] The server receives the image sent by the user, temporarily stores it in storage, and returns a response confirming receipt to the terminal.

[1682] Step 5:

[1683] The server runs the OCR engine

[1684] The server runs an optical character recognition (OCR) engine on the received image to extract text data from the image. For example, if the server uses the Tesseract OCR engine, it calls its API to process the data.

[1685] Step 6:

[1686] The server performs the layout analysis

[1687] The server analyzes the document layout based on the location information of the extracted text data, in order to identify the structure of tables and the positions of input fields, etc., and obtain data for generating a digital input form.

[1688] Step 7:

[1689] The server generates the digital input form

[1690] The server generates a digital input form based on the analysis results, which has the same layout (position of input fields, item names, etc.) as the original paper document.

[1691] Step 8:

[1692] The server encodes the input form

[1693] The server encodes the generated digital input form in a format such as JSON or HTML and prepares it to be sent to the user's device.

[1694] Step 9:

[1695] The server sends the input form to the terminal

[1696] The server transmits the encoded digital input form data to the user's terminal, including the transmitted data as a response.

[1697] Step 10:

[1698] The device receives the input form

[1699] The terminal receives and decodes the digital input form data sent from the server, thereby preparing to display the digital input form on the terminal.

[1700] Step 11:

[1701] The device displays the input form

[1702] The terminal displays the received digital input form on the screen so that the user can fill it in. The user interface is designed to be intuitive.

[1703] Step 12:

[1704] The user enters data into the input form

[1705] The user inputs necessary data into the digital input form displayed on the terminal, such as product name, quantity, and price.

[1706] Step 13:

[1707] User saves or submits input data

[1708] After completing the data entry, the user presses the "Save" or "Send" button. If the "Save" button is pressed, the data is temporarily saved on the terminal, and if the "Send" button is pressed, the data is sent to the server.

[1709] Step 14:

[1710] The device sends the input data to the server

[1711] The terminal sends the input data to the server via an HTTP request, encrypting the data if necessary.

[1712] Step 15:

[1713] The server receives and stores the data

[1714] The server validates the data received from the devices and stores it in a database for further processing and report generation.

[1715] Step 16:

[1716] The server performs data analysis and report generation

[1717] The server performs statistical processing and generates reports based on the stored data, such as aggregating sales data and generating monthly reports.

[1718] Through these steps, paper documents can be digitized quickly and accurately, resulting in more efficient operations and accurate data management.

[1719] Example 1

[1720] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1721] Conventional information collection and management systems required a huge amount of manual work to digitize paper documents, resulting in a lack of efficiency. Furthermore, manual data collection and management entailed the risk of input errors and data leaks, making it unreliable. Furthermore, there were insufficient methods for accurately digitizing paper documents and managing them electronically.

[1722] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1723] In this invention, the server includes means for a user to take an image of a document with an image input device and upload the image to a data processing device, means for the data processing device to receive the image and extract character data from the image using optical character recognition means, means for the data processing device to analyze the extracted character data and the layout of the image and automatically generate a digital input form, means for transmitting the generated digital input form to a user interface for the user and displaying it on the user interface, and means for transmitting data entered by the user on the user interface to the data processing device and storing it in a database, thereby enabling rapid and accurate digitization of paper documents and efficient information collection and management.

[1724] An "image input device" is a device with a camera function, including a smartphone or tablet terminal.

[1725] "Document" refers to information recorded on paper, including bills, receipts, medical records, etc.

[1726] A "data processing device" is a computer system for processing and analyzing image data and text data.

[1727] "Optical character recognition" means software or hardware technology for converting character information in an image into digital text, such as an OCR engine.

[1728] "Character data" is character information in digital form extracted by optical character recognition means.

[1729] "Layout" refers to the position information of character data within an image, including the layout of a document and the position of fields.

[1730] A "digital input form" is an electronic input screen generated based on analyzed character data and layout information.

[1731] A "user interface" refers to the display elements of a terminal screen or application that a user operates and uses to input data.

[1732] "Database" means a structured computer system for storing and managing processed data, including relational database systems.

[1733] "Encryption" means a technique for securely transforming data for the purpose of protecting the data during transmission.

[1734] This invention is a system that digitizes paper documents, eliminates the need for manual data entry, and enables efficient and accurate information collection. This system uses the following hardware and software:

[1735] This system starts with the user taking a photo of a paper document using an image input device (such as a smartphone or tablet), allowing the user to capture paper documents such as invoices, receipts, and medical records as image data.

[1736] Next, the user uploads the captured image to the data processing device. Using a dedicated application, the user selects an image and taps the upload button. The application then sends the image data to the server via an HTTP POST request.

[1737] The server first receives the image data and temporarily stores it. Then, the server launches an OCR engine (e.g., Tesseract) to extract the text data from the image. Specifically, it calls the OCR engine's API, reads the image file, and converts the text data into a digital format. For example, to use Tesseract OCR, run a command like "tesseract image.png output -l eng --psm 1."

[1738] The server then performs layout analysis based on the positional information of the character data. This analysis identifies the position of each text field and automatically generates a digital input form that is displayed in a format similar to the layout of the original paper document.

[1739] The server sends this digital input form to the user's device, specifically, it is encoded in HTML or JSON format and sent as an HTTP response, and the user's device receives this data and displays it on the application.

[1740] The user enters the required data into the digital input form displayed on the terminal, such as the invoice item name, quantity, amount, etc. After completion, the user presses the "Submit" button to send the data to the server, which then sends the data back to the server via an HTTP POST request.

[1741] The server stores the received data in a database using a relational database management system (e.g., MySQL or PostgreSQL) to store and manage the data accurately and efficiently. For example, it executes an INSERT statement to insert the data into the database.

[1742] Specific examples

[1743] Retailer Use

[1744] For example, this system is very useful for retailers who want to digitize their daily sales data. A retailer's employee takes a photo of each customer's receipt with their smartphone and uploads the image to a server using an application. The server analyzes the image and generates a digital receipt entry form. The employee enters the data on the device and presses the "send" button to send the data to the server. The server stores the received data in a database, making it easy to digitize sales data.

[1745] Use of medical facilities

[1746] In medical facilities, patient medical records are often kept on paper. By using this system, medical records are digitized and managed efficiently. Medical staff take photos of paper medical records and upload them to the system using a dedicated application. The server analyzes the images and generates a digital medical record entry form. Staff enter the necessary data into the form on their device, press the "send" button, and send it to the server, thereby advancing the digitization of medical records.

[1747] Prompt Sentence Examples

[1748] "Please explain the system a retailer uses to digitize sales data. The system consists of image capture, upload, OCR analysis, input form generation, data entry, and storage."

[1749] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1750] Step 1: Take an image

[1751] The user takes an image of a paper document using an image input device (smartphone or tablet). The user launches a camera app, adjusts the camera position so that the entire document fits in the image, and presses the capture button. The input for this step is the paper document, and the output is image data (JPEG or PNG format).

[1752] Step 2: Upload your images to the system

[1753] The user launches the dedicated application, selects the captured image, and taps the upload button. The application sends the selected image data to the server as an HTTP POST request. The input of this step is the image data, and the output is the image data sent to the server.

[1754] Step 3: Receive and save the image

[1755] The server receives the image data sent from the user's device and temporarily stores it. Specifically, the server receives the image data using XMLHttpRequest or Fetch API and stores it in a local temporary directory. The input to this step is the image data as an HTTP POST request, and the output is an image file stored in the server's temporary directory.

[1756] Step 4: Perform Optical Character Recognition (OCR)

[1757] The server invokes an OCR engine (e.g., Tesseract) on the stored image. Specifically, it calls the OCR engine's API to analyze the image file and extract the text data from the image. For example, when using Tesseract, execute a command like "tesseract image.png output -l eng --psm 1". The input for this step is the image file, and the output is digital text data.

[1758] Step 5: Perform layout analysis

[1759] The server performs layout analysis based on the positional information of the character data extracted by OCR. Specifically, it analyzes the coordinate information of each text field and executes an algorithm to identify the document layout. The input to this step is the character data and its positional information, and the output is a digital input form based on the layout information.

[1760] Step 6: Generate a digital input form

[1761] The server generates a digital input form based on the layout analysis results. Specifically, it dynamically generates a form template and embeds the analysis results. The input for this step is layout information and character data, and the output is the HTML or JSON data of the digital input form.

[1762] Step 7: Submitting the digital input form

[1763] The server sends the generated digital input form to the user's device. Specifically, it sends data encoded in HTML or JSON format as an HTTP response. The input of this step is the data of the digital input form, and the output is the digital input form displayed on the user's device.

[1764] Step 8: Enter your data

[1765] The user enters data into a digital input form displayed on the terminal display. Specifically, data is entered into form elements such as text boxes and drop-down lists. The input of this step is the user's input data, and the output is the data entered into the form.

[1766] Step 9: Sending Data

[1767] After the user has completed entering data, they click the "Submit" button to send the input data to the server. Specifically, the form data is sent to the server as an HTTP POST request. The input of this step is the data entered by the user, and the output is the data sent to the server.

[1768] Step 10: Save your data

[1769] The server stores the received data in a database. Specifically, it uses a relational database management system (e.g., MySQL or PostgreSQL) to add records to the database by executing SQL INSERT statements. The input to this step is the data sent to the server, and the output is the records stored in the database.

[1770] (Application example 1)

[1771] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1772] When digitizing paper documents, manual data entry is extremely time-consuming and labor-intensive, and prone to human error. In addition, factories need to manage a wide range of paper documents, making efficient digitization a necessity. Furthermore, mechanical methods for converting paper data into digital format are not being used effectively.

[1773] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1774] In this invention, the server includes means for a user to photograph an image of a paper document and upload the image to the system, means for the server to receive the image and extract text data from the image using optical character recognition means, means for the server to analyze the extracted text data and the layout of the image and automatically generate a digital input form, means for sending the generated digital input form to the user's terminal and displaying it on the terminal, means for sending data entered by the user on the terminal to the server and storing it in a database, means for a robot to photograph a paper document and upload the image to the server, means for the robot to display the generated digital input form and for an operator to enter data, and means for sending the entered data to the server and storing it in a database, thereby enabling efficient and accurate digitization of paper documents.

[1775] "User" means a person or entity that uses the System to digitize paper documents.

[1776] "Paper documents" refers to documents or documents printed or recorded on paper, such as invoices, receipts, manuals, and checklists.

[1777] An "image" is digital visual information captured by a camera of a paper document.

[1778] In the present invention, the term "system" refers to a set of processes and devices for digitizing paper documents.

[1779] A "server" is a computing resource that analyzes received images, generates digital input forms, and stores data.

[1780] "Optical character recognition" is a technology for analyzing text within an image and converting it into digital character data.

[1781] A "digital input form" is a digital form into which a user can input data, constructed based on extracted text data and layout analysis.

[1782] A "robot" is a mechanical device that automatically photographs paper documents in factories and other work sites and uploads the images to the system.

[1783] A "database" is a system that stores input data and allows it to be managed and searched as needed.

[1784] A "worker" is a person whose role is to operate a robot or enter data into a digital input form.

[1785] MODE FOR CARRYING OUT THE INVENTION

[1786] This invention is a system that digitizes paper documents and enables efficient and accurate information collection. The system starts when a user takes a photo of a paper document and uploads the image to a server using a robot or terminal.

[1787] The server first receives the image data and extracts the text data from the image using optical character recognition (OCR). Specifically, an OCR library such as Tesseract is used. The extracted text data and the image layout are then analyzed to generate a digital input form. OpenCV is used to preprocess the image, removing noise and detecting contours, enabling accurate text extraction.

[1788] The generated digital input form is sent to the user's terminal or robot and displayed. The user or worker enters data on these terminals or devices, and the data is then sent back to the server and stored in a database. This makes data entry from paper more efficient and reduces input errors.

[1789] This system is particularly effective in factories. The process of factory robots automatically photographing and digitizing paper checklists dramatically improves on-site data management. Furthermore, by analyzing the layout of the input form, a digital form is created that closely matches the format of the original paper document, allowing workers to enter data without any discomfort.

[1790] A concrete example would be the use of a checklist in a factory. For example, if there is a checklist like this:

[1791] Machine name: XYZ product

[1792] Check item: Oil level

[1793] Inspection result: Pass Fail

[1794] The robot takes a photo of this checklist and uploads it to the system. The text is analyzed by Tesseract and sent to the server, where an input form is generated based on the layout analysis.

[1795] Examples of prompts include:

[1796] Generate a digital input form from the following text:

[1797] Machine name: XYZ product

[1798] Check item: Oil level

[1799] Inspection result: Pass Fail

[1800] The generated form should have the following format:

[1801] {

[1802] "formFields": [

[1803] {"label": "machine name", "type": "text"},

[1804] {"label": "Inspection items", "type": "text"},

[1805] {"label": "Inspection result", "type": "radio", "options": ["Pass", "Fail"]}

[1806] ]

[1807] }

[1808] Using these prompts, accurate digital input forms are automatically generated. The data is then stored in a database for further processing and analysis. This system allows for the rapid and accurate digitization of paper-based data.

[1809] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1810] Step 1:

[1811] A user takes an image of a paper document.

[1812] Input: Paper documents

[1813] Output: Captured image data (e.g., JPEG image file)

[1814] Specific operation: The user uses the camera on their smartphone or tablet to take a picture of the checklist or manual.

[1815] Step 2:

[1816] The robot uploads the captured images to the system.

[1817] Input: Photographed image data

[1818] Output: Image data sent to the server

[1819] Specific operation: Images taken by the camera mounted on the robot are transferred to the built-in control unit and uploaded to a server via the network.

[1820] Step 3:

[1821] The server receives the uploaded images and temporarily stores them in a database.

[1822] Input: Uploaded image data

[1823] Output: Temporarily saved image data

[1824] Specific operation: The server receives the HTTP request and saves the image data in a designated folder for temporary storage.

[1825] Step 4:

[1826] The server performs OCR (optical character recognition) and extracts text data from the image data.

[1827] Input: Temporarily saved image data

[1828] Output: Extracted text data

[1829] How it works: The server uses an OCR library such as Tesseract to analyze the letters and numbers in the image and convert them into digital text.

[1830] Step 5:

[1831] The server performs layout analysis and generates a digital input form.

[1832] Input: Extracted text data and image layout information

[1833] Output: Generated digital input form (e.g. JSON format)

[1834] Specific operation: The server uses image processing libraries such as OpenCV to analyze the positional information of the text and generate a digital input form that closely resembles the layout of paper media.

[1835] Step 6:

[1836] The server sends the generated digital input form to the terminal or robot.

[1837] Input: Generated digital input form

[1838] Output: Submitted digital input form

[1839] Specific operation: The server sends the generated form to the terminal or robot as an HTTP response.

[1840] Step 7:

[1841] Users or workers enter data into a digital input form displayed on a terminal or robot display.

[1842] Input: Digital input form

[1843] Output: The data entered (e.g., the value of a form field)

[1844] Specific operation: The user or worker uses the touch panel or keyboard of the terminal or robot to enter the required data into the displayed form.

[1845] Step 8:

[1846] The terminal or robot sends the entered data to the server.

[1847] Input: Data entered

[1848] Output: Data sent to the server

[1849] Specific operation: When a user or worker presses the "Send" button, the entered data is sent to the server as an HTTP request.

[1850] Step 9:

[1851] The server stores the received data in a database.

[1852] Input: Data sent to the server

[1853] Output: Data stored in the database

[1854] What happens: The server converts the received data into an appropriate format and inserts it into a database, where it can be used for later analysis, reporting, etc.

[1855] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1856] The present invention is a system that digitizes paper documents, eliminates the need for manual data entry, and enables efficient and accurate information collection, and further includes a means for recognizing the user's emotions and reflecting the results in the system. Specific embodiments of the various means are described below.

[1857] Program processing overview

[1858] Image upload

[1859] 1. The user takes a picture on the device

[1860] Users use the camera on their smartphone or tablet to take a picture of a paper document (such as an invoice or receipt) using either a dedicated camera app or the device's built-in camera app.

[1861] 2. User uploads image to the system

[1862] The user launches a dedicated application, selects the captured image, and presses the "upload" button to send the image to the system.

[1863] Image processing, analysis and emotion recognition

[1864] 3. The server receives the image

[1865] The server receives the image sent from the user's device, temporarily stores it in storage, and returns a response confirming receipt to the device.

[1866] 4. The server runs the OCR engine

[1867] The server runs an optical character recognition (OCR) engine on the received image to extract text data from the image. For example, if the server uses the Tesseract OCR engine, it calls its API to process the data.

[1868] 5. The server performs layout analysis

[1869] The server analyzes the layout of the document based on the location information of the text data, and as a result, the digital input form is reproduced with the same layout as the paper document.

[1870] 6. The server runs the emotion engine

[1871] The server analyzes the user's image and input actions and activates an emotion engine that recognizes the user's emotions. For example, it uses an emotion recognition API to analyze emotions from the user's facial expressions and input speed.

[1872] Input form generation and display

[1873] 7. The server generates the digital input form

[1874] The server generates a digital input form based on the analysis results, which has the same layout (position of input fields, item names, etc.) as the original paper document.

[1875] 8. The server encodes the input form

[1876] The server encodes the generated digital input form in a format such as JSON or HTML and prepares it to be sent to the user's device.

[1877] 9. The server sends the input form to the device

[1878] The server transmits the encoded digital input form data to the user's terminal, including the transmitted data as a response.

[1879] 10. The device receives the input form

[1880] The terminal receives and decodes the digital input form data sent from the server, thereby preparing to display the digital input form on the terminal.

[1881] 11. The device displays the input form

[1882] The terminal displays the received digital input form on the screen so that the user can fill it in. The user interface is designed to be intuitive.

[1883] Data entry and transmission and emotional feedback

[1884] 12. User enters data into input form

[1885] The user inputs necessary data into the digital input form displayed on the terminal, such as product name, quantity, and price.

[1886] 13. Feedback from the Emotion Engine

[1887] The emotion engine dynamically changes the design and message as you type, reflecting your emotions. For example, if you're feeling stressed, it will present a relaxing message or color tone.

[1888] 14. Save or submit user-entered data

[1889] After completing the data entry, the user presses the "Save" or "Send" button. If the "Save" button is pressed, the data is temporarily saved on the terminal, and if the "Send" button is pressed, the data is sent to the server.

[1890] 15. The device sends the input data to the server

[1891] When the terminal transmits the input data to the server via an HTTP request, the terminal encrypts the data as necessary.

[1892] 16. The server receives and stores the data

[1893] The server validates the data received from the devices and stores it in a database for further processing and report generation.

[1894] 17. The server performs data analysis and report generation

[1895] The server performs statistical processing and generates reports based on the stored data, such as aggregating sales data and generating monthly reports.

[1896] Specific examples

[1897] Retailer Use

[1898] For example, this system would be extremely useful for retailers to easily digitize their daily sales data. Retailers' staff members take photos of each customer's receipts with their smartphones and upload them. The server analyzes the images and generates a digital receipt entry form. The staff member enters the data on the device and stores it on the server, making it easy to digitize sales data. In addition, the emotion engine detects the staff member's stress and provides feedback to help them relax.

[1899] Use of medical facilities

[1900] In medical facilities, patient medical records are often kept on paper. By using this system, medical records are digitized and managed efficiently. Medical staff take photos of paper medical records and upload them to the system. The server analyzes the images and generates a digital medical record entry form. Staff enter the necessary data into the form and save it on the server, further increasing the digitization of medical records. An emotion engine detects staff fatigue and provides refreshing messages, improving work efficiency and comfort.

[1901] These processes enable rapid and accurate digitization of paper documents, improving work efficiency and ensuring accurate data management. Furthermore, the use of an emotion engine is expected to reduce user stress and improve work efficiency.

[1902] The processing flow will be explained below.

[1903] Step 1:

[1904] The user takes a picture on the device

[1905] Users use the camera on their smartphone or tablet to take a picture of a paper document (such as an invoice or receipt) using either a dedicated camera app or the device's built-in camera app.

[1906] Step 2:

[1907] User uploads an image to the system

[1908] The user launches a dedicated application, selects the captured image, and presses the "upload" button to send the image to the system.

[1909] Step 3:

[1910] The device sends the image to the server

[1911] The device uses an HTTP request (such as a POST request) to send the captured image to the server, compressing or encrypting the image data as necessary.

[1912] Step 4:

[1913] The server receives the image

[1914] The server receives the image sent by the user, temporarily stores it in storage, and returns a response confirming receipt to the terminal.

[1915] Step 5:

[1916] The server runs the OCR engine

[1917] The server runs an optical character recognition (OCR) engine on the received image to extract text data from the image. For example, if the server uses the Tesseract OCR engine, it calls its API to process the data.

[1918] Step 6:

[1919] The server performs the layout analysis

[1920] The server analyzes the layout of the document based on the location information of the text data, and as a result, the digital input form is reproduced with the same layout as the paper document.

[1921] Step 7:

[1922] The server runs the emotion engine

[1923] The server analyzes the user's image and input actions and activates an emotion engine that recognizes the user's emotions. For example, it uses an emotion recognition API to analyze emotions from the user's facial expressions and input speed.

[1924] Step 8:

[1925] The server generates the digital input form

[1926] The server generates a digital input form based on the analysis results, which has the same layout (position of input fields, item names, etc.) as the original paper document.

[1927] Step 9:

[1928] The server encodes the input form

[1929] The server encodes the generated digital input form in a format such as JSON or HTML and prepares it to be sent to the user's device.

[1930] Step 10:

[1931] The server sends the input form to the terminal

[1932] The server transmits the encoded digital input form data to the user's terminal, including the transmitted data as a response.

[1933] Step 11:

[1934] The device receives the input form

[1935] The terminal receives and decodes the digital input form data sent from the server, thereby preparing to display the digital input form on the terminal.

[1936] Step 12:

[1937] The device displays the input form

[1938] The terminal displays the received digital input form on the screen so that the user can fill it in. The user interface is designed to be intuitive.

[1939] Step 13:

[1940] The user enters data into the input form

[1941] The user inputs necessary data into the digital input form displayed on the terminal, such as product name, quantity, and price.

[1942] Step 14:

[1943] Feedback from the Emotion Engine

[1944] The server's emotion engine monitors the user's facial expressions and behavior while they are typing and evaluates them in real time. For example, if the user is feeling stressed, a relaxing message or color tone will be dynamically displayed on the input form.

[1945] Step 15:

[1946] User saves or submits input data

[1947] After completing the data entry, the user presses the "Save" or "Send" button. When the "Save" button is pressed, the data is temporarily saved on the terminal, and when the "Send" button is pressed, the data is sent to the server.

[1948] Step 16:

[1949] The device sends the input data to the server

[1950] The terminal sends the input data to the server via an HTTP request, encrypting the data if necessary.

[1951] Step 17:

[1952] The server receives and stores the data

[1953] The server verifies the data received from the devices and stores it in a database for further processing and report generation.

[1954] Step 18:

[1955] The server performs data analysis and report generation

[1956] The server performs statistical processing and generates reports based on the stored data, such as aggregating sales data and generating monthly reports.

[1957] Through these steps, paper documents can be digitized quickly and accurately, improving work efficiency and data management. Furthermore, the use of an emotion engine is expected to reduce user stress and improve work efficiency.

[1958] Example 2

[1959] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1960] Digitizing paper documents requires manual data entry, which is time-consuming and labor-intensive. Furthermore, traditional systems often have difficulty accurately reproducing the layout of documents. Furthermore, there is no feedback based on the user's emotions or state, resulting in a lack of consideration for user stress and work efficiency.

[1961] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for extracting text data from an image using optical character recognition means, means for automatically generating a digital input form by analyzing the extracted text data and the layout of the image, and means for analyzing the user's image and input actions, recognizing the user's emotions, and providing feedback. This makes it possible to efficiently and accurately digitize paper documents, further reducing user stress and improving work efficiency.

[1962] "User" means any person or legal entity that uses the System to digitize paper documents.

[1963] A "paper document" is a document containing information printed or handwritten on paper.

[1964] "Means for taking images" refers to a method or device for taking images of paper documents using a device equipped with a camera function (such as a smartphone or tablet).

[1965] "System" means a set of devices and programs, including servers and terminals, for digitizing paper documents and processing and storing data.

[1966] A "server" is a computer that receives images, processes data, and stores it.

[1967] "Optical character recognition" refers to technology or equipment that analyzes character information in an image and extracts it as text data. An example of this is an OCR (optical character recognition) engine.

[1968] "Text data" refers to character information extracted by optical character recognition means.

[1969] A "layout analysis means" is a technique or device that analyzes the structure and layout of a document based on the positional information of text data within an image.

[1970] A "digital input form" is an input form that is reproduced in a digital environment based on a paper document.

[1971] "Means for providing feedback" refers to technologies or devices that provide appropriate responses or advice in real time based on the user's emotions and input actions.

[1972] "Encryption means" refers to a technique or device that encrypts data to ensure data security.

[1973] A "database" is a storage device or system for storing and managing data received by a server.

[1974] "Emotion recognition means" refers to technology or devices that analyze and evaluate a user's emotional state based on the user's image and input actions.

[1975] This system aims to digitize paper documents, improve work efficiency, and organize data accurately. Furthermore, it analyzes user emotions and provides feedback to improve usability. This system operates in cooperation with users, terminals, and servers.

[1976] Hardware and software used

[1977] Hardware: Smartphones, tablets, and server computers used by users

[1978] Software: Camera app (dedicated app or device-specific app), OCR engine (e.g., Tesseract), emotion recognition API, database system

[1979] Program processing overview

[1980] 1. The user takes a picture on the device

[1981] The user uses the camera on their smartphone or tablet to take a picture of a paper document (e.g., an invoice or receipt) using a dedicated app or the device's standard camera app.

[1982] 2. User uploads image to the system

[1983] The user launches a dedicated application, selects the captured image, and presses the "upload" button to send the image to the system.

[1984] 3. The server receives the image

[1985] The server receives the image sent from the user's device, temporarily stores it in storage, and returns a response confirming receipt to the device.

[1986] 4. The server runs the OCR engine

[1987] The server uses an optical character recognition (OCR) engine to extract text data from the image. Specifically, it calls the API of the Tesseract OCR engine to obtain the character information in the image as text data.

[1988] 5. The server performs layout analysis

[1989] The server analyzes the layout based on the location information of the extracted text data and generates a digital input form, which creates a digital form that retains the same layout as the paper document.

[1990] 6. The server runs the emotion engine

[1991] The server analyzes the user's image and input actions to activate an emotion engine that recognizes emotions. Specifically, it uses an emotion recognition API to analyze the user's emotions based on their facial expressions and input speed, and provides feedback.

[1992] 7. The server generates and submits the digital input form

[1993] The server generates a digital input form based on the analysis results, encodes this form in JSON or HTML format, and sends it to the user's device.

[1994] 8. The device receives and displays the input form

[1995] The terminal receives the digital input form data sent from the server, decodes it, and displays it, allowing the user to enter the required data into the digital input form.

[1996] 9. The device sends the input data to the server

[1997] After the user has completed the data entry, the data is sent from the terminal to the server, where it is encrypted if necessary.

[1998] 10. The server receives and stores the data

[1999] The server validates the received data and stores it securely in a database for later statistical processing and report generation.

[2000] 11. The server performs data analysis and report generation

[2001] The server performs statistical processing and generates reports based on the stored data, such as compiling sales data and generating monthly reports.

[2002] Specific examples

[2003] For retailers

[2004] This system is extremely useful for retailers digitizing their daily sales data. When a salesperson takes a photo of a customer's receipt with their smartphone and uploads it, the server analyzes the image and generates a digital receipt entry form. The salesperson enters the data on the device and sends it to the server. The system uses an emotion engine to detect the salesperson's stress and provides a relaxing message.

[2005] For medical facilities

[2006] Medical facilities use this system to digitize paper medical records. When medical staff take a photo of a paper medical record and upload it to the system, the server analyzes the image and generates a digital input form. The staff enters data into the form and sends it to the server. An emotion engine detects staff fatigue and displays a refresh message. This allows for efficient management of medical records and improves work comfort.

[2007] In this way, the system of the present invention not only efficiently and accurately digitizes paper documents, but also provides technical means for reducing user stress and improving work efficiency.

[2008] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2009] Step 1:

[2010] The user takes a photo of a paper document

[2011] Input: Paper documents (e.g. invoices, receipts)

[2012] How it works: The user uses the built-in camera on their smartphone or tablet to take a photo of a paper document using a dedicated app or the device's standard camera app.

[2013] Output: Captured image data

[2014] Step 2:

[2015] A user uploads an image to the system

[2016] Input: Captured image data

[2017] How it works: The user launches the dedicated application, selects the captured image, and presses the "Upload" button, which sends the image data to the server via the Internet.

[2018] Output: Image data sent to the server

[2019] Step 3:

[2020] The server receives the image

[2021] Input: Image data sent by the user

[2022] Operation: The server temporarily stores the received image data in storage and returns a response confirming receipt to the user device.

[2023] Output: Image data stored in storage, receipt confirmation response

[2024] Step 4:

[2025] The server runs the OCR engine

[2026] Input: Received and stored image data

[2027] How it works: The server uses an optical character recognition (OCR) engine (e.g., Tesseract) to extract character information from the image data as text data.

[2028] Output: Extracted text data

[2029] Step 5:

[2030] The server performs the layout analysis

[2031] Input: Extracted text and image data

[2032] How it works: The server analyzes the location information of the text data and the layout of the images, and automatically generates a digital input form based on the layout of the paper medium.

[2033] Output: The generated digital input form

[2034] Step 6:

[2035] The server runs the emotion engine

[2036] Input: User images, input data, and action data

[2037] How it works: The server uses the emotion recognition API to analyze the user's facial expressions and input speed to recognize their emotions. Based on the analysis results, it generates feedback content for the user.

[2038] Output: User sentiment analysis results, feedback content

[2039] Step 7:

[2040] The server generates and submits the digital input form

[2041] Input: Digital input form, feedback content

[2042] How it works: The server encodes the generated digital input form in JSON or HTML format and sends it to the user's device, optionally incorporating feedback into the submitted data.

[2043] Output: Encoded digital input form data, feedback data

[2044] Step 8:

[2045] The device receives and displays the input form.

[2046] Input: Digital input form data sent from the server

[2047] Operation: The user terminal receives the image data, decodes it, and displays an input form. The user can then enter the required data into the form.

[2048] Output: Digital input form displayed on the terminal

[2049] Step 9:

[2050] The user enters data into the form

[2051] Input: Digital input forms, data entered by users

[2052] How it works: The user enters data such as product name, quantity, and price into a digital input form displayed on the terminal.

[2053] Output: Data entered by the user

[2054] Step 10:

[2055] The device sends the input data to the server

[2056] Input: Data entered by the user

[2057] How it works: When a user presses the "Save" or "Submit" button, the device sends the input data to the server via an HTTP request, encrypting the data if necessary.

[2058] Output: Input data sent to the server

[2059] Step 11:

[2060] The server receives and stores the data

[2061] Input: Input data sent from the terminal

[2062] Operation: The server stores the received data in a database for later statistical processing and report generation.

[2063] Output: Data stored in the database

[2064] Step 12:

[2065] The server performs data analysis and report generation.

[2066] Input: Data stored in the database

[2067] How it works: The server processes the stored data and generates statistics and reports, such as aggregating sales data and generating monthly reports.

[2068] Output: easy-to-read statistics and reports

[2069] (Application example 2)

[2070] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2071] In food delivery operations, digitizing handwritten order forms and delivery confirmations has become a problem, resulting in time, mistakes, and stress for delivery workers. Furthermore, managing paper documents is complicated, creating a need for efficient data management. Furthermore, there is a lack of methods to reduce the workload and manage stress for delivery workers.

[2072] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for a user to take an image of a paper document and upload the image to the system; means for the server to receive the image and extract text data from the image using optical character recognition means; means for the server to analyze the extracted text data and the image layout and automatically generate a digital input form; means for the server to send the generated digital input form to the user's terminal and display it on the terminal; means for the server to send data entered by the user on the terminal and store it in a database; and means for the server to recognize the user's emotions using image analysis means and provide feedback based on the emotions to the terminal. This allows delivery personnel to efficiently and accurately digitize paper documents and manage stress in real time.

[2073] Definitions of important words

[2074] "User" means a person or organization that uses a system to digitize paper documents.

[2075] A "paper document" is a document containing information printed or handwritten on paper.

[2076] An "image" is visual data captured using a device such as a camera.

[2077] "Upload" is the act of sending data from a user's device to a server.

[2078] "System" refers to a set of technological means, including computers and software.

[2079] A "server" is a computer system that centrally receives, processes, and stores data.

[2080] "Optical character recognition" is a technology that analyzes text within an image and converts it into digital data.

[2081] "Text data" is digital information expressed as a string of characters.

[2082] "Layout" refers to information about the arrangement of text, images, etc. in a paper document.

[2083] "Digital input form" means an electronic form for entering data from a digitized document.

[2084] A "terminal" is an electronic device operated by a user, such as a smartphone or tablet.

[2085] "Image analysis means" is a technique for extracting specific information from an image.

[2086] "Emotion recognition" is a technology that analyzes emotions from a user's facial expressions and behavior.

[2087] "Feedback" refers to the information or response that a system provides to a user.

[2088] A "database" is a system for efficiently storing and managing digital data.

[2089] Description (Mode for carrying out the invention)

[2090] The present invention provides a system that allows users to digitize paper documents and manage information efficiently and accurately. Specific embodiments of the present invention will be described below.

[2091] Hardware and Software Configuration

[2092] Hardware:

[2093] Device: An electronic device such as a smartphone or tablet.

[2094] Server: A computer system that receives, processes, and stores data.

[2095] software:

[2096] Flask: Implementing server programs using a web framework.

[2097] PaddleOCR: An optical character recognition engine for extracting text from images.

[2098] FER: Analyzes emotions from facial expressions in images through an emotion recognition engine.

[2099] OpenCV: Use the image processing library to analyze and process images.

[2100] System Operation

[2101] The server performs the following process.

[2102] 1. Receiving and saving images: This receives images of paper documents taken by users on their devices and temporarily saves them in storage. This is implemented using the Flask framework.

[2103] 2. Text extraction using OCR: PaddleOCR is used to extract text data from images. The OCR engine processes the received image and returns the extracted text data to the server.

[2104] 3. Layout analysis: The text data extracted by OCR is analyzed to recreate the same layout as the original paper document. This processing is performed on the server side, allowing the generation of a digital input form.

[2105] 4. Emotion Recognition: FER is used to analyze the user's emotions. Based on facial expressions in the image and input speed, emotions are analyzed and feedback corresponding to the emotion is provided to the device.

[2106] 5. Sending and displaying the digital input form: After the analysis is complete, the generated digital input form is encoded in JSON or HTML format and sent to the user's device, where it displays the received data and allows the user to enter information.

[2107] Specific examples

[2108] For example, this system is extremely useful when food delivery workers need to digitize handwritten order forms. The delivery worker takes a photo of the order form with their smartphone and uploads it to a server using a dedicated application. The server receives the image, extracts text data using OCR, and generates a digital input form by analyzing the layout. The generated digital input form is then sent to a terminal, where the delivery worker can enter the necessary information. In addition, an emotion recognition engine can provide relaxing messages or feedback if the delivery worker's stress level is high.

[2109] Example prompt sentence:

[2110] "Please recognize characters in an image and convert them into digital text. Also, please analyze the emotions of faces in the image and return the results in JSON format."

[2111] In this way, paper documents can be digitized quickly and accurately, reducing the workload of delivery personnel and improving work efficiency.

[2112] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2113] Program processing steps

[2114] Step 1:

[2115] The user (delivery person) takes a picture of the paper order form using the camera on their smartphone. The image is temporarily saved in the device's storage.

[2116] Input: Paper purchase order

[2117] Output: Image file saved on device

[2118] Step 2:

[2119] The user launches the dedicated application, selects the image they have taken, and presses the "upload" button, which sends the image to the server.

[2120] Input: Image file saved on your smartphone

[2121] Output: Image data sent to the server

[2122] Step 3:

[2123] The server receives the image, temporarily stores it in storage, and returns a reception confirmation response to the device.

[2124] Input: Image data sent by the user

[2125] Output: Image file saved on the server and a receipt confirmation response to the device

[2126] Step 4:

[2127] The server runs an OCR engine (PaddleOCR) to extract text data from the received images.

[2128] Input: Image file stored on the server

[2129] Output: Extracted text data

[2130] Step 5:

[2131] The server analyzes the layout and generates a digital input form based on the positional information of the text data, recreating the same layout (position of input fields, item names, etc.) as the original paper document.

[2132] Input: Text data extracted by OCR

[2133] Output: Digital input form data

[2134] Step 6:

[2135] The server runs an emotion recognition engine (FER) to analyze the user's emotions from their facial expressions and input actions, and generates appropriate feedback and sends it to the user's device.

[2136] Input: Received image and user input action data

[2137] Output: User sentiment analysis results and feedback

[2138] Step 7:

[2139] The server encodes the generated digital input form in JSON or HTML format and sends it to the user's device. The device then decodes the received digital input form data and displays it on the screen.

[2140] Input: Digital input form data

[2141] Output: A digital input form displayed on the user's device

[2142] Step 8:

[2143] The user enters the necessary data into the digital input form displayed on the terminal, such as product information, quantity, and price.

[2144] Input: User-entered data

[2145] Output: Input digital data

[2146] Step 9:

[2147] The user presses a button to save or send the input data. If the save button is pressed, the data is temporarily saved on the terminal, and if the send button is pressed, the data is sent to the server.

[2148] Input: User input data and button operations

[2149] Output: Data stored on the device or sent to a server

[2150] Step 10:

[2151] The server stores the received data in a database and performs statistical processing and generates reports as needed.

[2152] Input: Transmitted digital data

[2153] Output: Data stored in the database and generated reports

[2154] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2155] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2156] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2157] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2158] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2159] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2160] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2161] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2162] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2163] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2164] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2165] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2166] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2167] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2168] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2169] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2170] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2171] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2172] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2173] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2174] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2175] The following is further disclosed regarding the above embodiment.

[2176] (Claim 1)

[2177] A means for a user to take an image of a paper document and upload the image to the system;

[2178] a server receiving the image and extracting text data within the image using optical character recognition;

[2179] a server that analyzes the extracted text data and image layout to automatically generate a digital input form;

[2180] means for transmitting the generated digital input form to a user's terminal and displaying it on the terminal;

[2181] means for transmitting data input by a user at said terminal to a server, and for said server to store the data in a database;

[2182] A system including:

[2183] (Claim 2)

[2184] 10. The system of claim 1, further comprising means for analyzing column positions and structure in order to reproduce the layout of the original paper document in generating the digital input form.

[2185] (Claim 3)

[2186] 10. The system of claim 1, further comprising means for protecting image data using encryption means when the terminal transmits the captured image to a server.

[2187] "Example 1"

[2188] (Claim 1)

[2189] a means for a user to take an image of a document using an image input device and upload the image to a data processing device;

[2190] a data processing device for receiving the image and extracting character data within the image using optical character recognition means;

[2191] a means for automatically generating a digital input form by a data processing device by analyzing the extracted character data and image layout;

[2192] means for transmitting the generated digital input form to a user's user interface and displaying it on said user interface;

[2193] means for transmitting data input by a user via the user interface to a data processing device, and for the data processing device to store the data in a database;

[2194] A system including:

[2195] (Claim 2)

[2196] 10. The system of claim 1, further comprising means for analyzing the position and structure of fields in generating the digital input form to recreate the layout of the original document.

[2197] (Claim 3)

[2198] 10. The system of claim 1, further comprising means for protecting image data using encryption means when the user interface transmits the captured image to a data processing device.

[2199] "Application Example 1"

[2200] (Claim 1)

[2201] A means for a user to take an image of a paper document and upload the image to the system;

[2202] a server receiving the image and extracting text data within the image using optical character recognition;

[2203] a server that analyzes the extracted text data and image layout to automatically generate a digital input form;

[2204] means for transmitting the generated digital input form to a user's terminal and displaying it on the terminal;

[2205] means for transmitting data input by a user at said terminal to a server, and for said server to store the data in a database;

[2206] A robot takes a photo of the paper document and uploads the image to a server.

[2207] a means for displaying the generated digital input form by the robot and for allowing a worker to input data;

[2208] means for transmitting the input data to a server, and for the server to store the data in a database;

[2209] A system including:

[2210] (Claim 2)

[2211] 10. The system of claim 1, further comprising means for analyzing column positions and structure in order to reproduce the layout of the original paper document in generating the digital input form.

[2212] (Claim 3)

[2213] 10. The system of claim 1, further comprising means for protecting image data using encryption means when the terminal transmits the captured image to a server.

[2214] "Example 2: Combining Emotion Engines"

[2215] (Claim 1)

[2216] A means for a user to take an image of a paper document and upload the image to the system;

[2217] a server receiving the image and extracting text data within the image using optical character recognition;

[2218] a server that analyzes the extracted text data and image layout to automatically generate a digital input form;

[2219] means for transmitting the generated digital input form to a user's terminal and displaying it on the terminal;

[2220] means for transmitting data input by a user at said terminal to a server, and for said server to store the data in a database;

[2221] A means for the server to analyze the user's image and input actions, recognize the user's emotions, and provide feedback;

[2222] A system including:

[2223] (Claim 2)

[2224] 10. The system of claim 1, further comprising means for analyzing column locations and structure to recreate the layout of the original paper document.

[2225] (Claim 3)

[2226] 10. The system of claim 1, further comprising means for protecting image data and input data transmitted from the terminal to the server using strong encryption means.

[2227] "Application example 2 when combining emotion engines"

[2228] (Claim 1)

[2229] A means for a user to take an image of a paper document and upload the image to the system;

[2230] a server receiving the image and extracting text data within the image using optical character recognition;

[2231] a server that analyzes the extracted text data and image layout to automatically generate a digital input form;

[2232] means for transmitting the generated digital input form to a user's terminal and displaying it on the terminal;

[2233] means for transmitting data input by a user at said terminal to a server, and for said server to store the data in a database;

[2234] a means for the server to recognize the user's emotion using an image analysis means and provide feedback based on the emotion to the terminal;

[2235] A system including:

[2236] (Claim 2)

[2237] 10. The system of claim 1, further comprising means for analyzing column positions and structure to recreate the layout of the original paper document in generating the digital input form.

[2238] (Claim 3)

[2239] 10. The system of claim 1, further comprising: means for protecting image data using information protection means when the terminal transmits the captured image to the server. [Explanation of symbols]

[2240] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for a user to take an image of a paper document and upload the image to the system; a server receiving the image and extracting text data within the image using optical character recognition; a server that analyzes the extracted text data and image layout to automatically generate a digital input form; means for transmitting the generated digital input form to a user's terminal and displaying it on the terminal; means for transmitting data input by a user at said terminal to a server, and for said server to store the data in a database; A system including:

2. The system of claim 1 , further comprising means for analyzing the position and structure of columns in order to reproduce the layout of the original paper document in generating the digital input form.

3. The system according to claim 1 , further comprising means for protecting image data using encryption means when the terminal transmits the captured image to a server.

Images