system

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving an image or PDF file of a bill; means for converting the received file into text data using OCR technology; means for analyzing the converted text data using a generation AI and extracting each item of the bill; means for matching issuer information of the extracted bill with an eligible bill issuer database; means for determining the bill as an eligible bill based on the matching result; and means for generating a response to a query from a user using the generation AI.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] With the invoice system that will be implemented from October 2023, the amount of tax deduction will change depending on whether or not a business is a qualified invoice issuer, which means that businesses will have to deal with an increased number of tasks, such as checking and storing invoices and verifying whether or not a business is a qualified invoice issuer. This will require businesses to spend a great deal of time and effort on these tasks, raising concerns that this will reduce business efficiency. [Means for solving the problem]

[0005] This invention provides a system that automates a series of tasks, from scanning invoices to determining whether they are qualified invoices, digitizing them, saving them, and responding to inquiries. Specifically, the proposed system includes a means for receiving invoice images or PDF files, a means for converting the received files into text data using OCR technology, a means for analyzing the converted text data using a generation AI and extracting each item on the invoice, a means for comparing the extracted invoice issuer information with a database of qualified invoice issuers, a means for determining whether the invoice is a qualified invoice based on the comparison results, a means for storing the determined invoices as electronic data on a cloud platform, and a means for generating responses to user inquiries using a generation AI. This system allows businesses to streamline invoice-related tasks and significantly reduce the effort and time required.

[0006] An "invoice" is a document that lists the amount to be paid in a transaction and a detailed breakdown of the amount.

[0007] An "image" is digital or analog visual data that represents an object or scene.

[0008] A "PDF file" is a file format for electronic documents saved in Portable Document Format.

[0009] "OCR technology" stands for Optical Character Recognition, and is a technology that extracts text from images and PDF files.

[0010] "Character data" refers to information expressed in text format.

[0011] "Generative AI" is artificial intelligence that generates, analyzes, and responds to data using techniques such as generative adversarial networks (GANs) and natural language processing (NLP).

[0012] "Analyzing" means examining data in detail to understand its meaning and structure.

[0013] An "item" is an element or content that is independently divided within information.

[0014] "Issuer information" refers to information such as the name, address, and contact details of the company or individual that issued the invoice.

[0015] The "Qualified Invoice Issuer Database" is a database that centrally manages information on businesses registered to issue qualified invoices.

[0016] "Matching" means comparing different data or information to see if they match.

[0017] To "judge" means to reach a conclusion based on certain criteria.

[0018] "Electronic data" means information stored in digital form.

[0019] A "cloud infrastructure" is a network of remote servers that can store and process data over the internet.

[0020] "Storing" means storing data for future reference or use.

[0021] An "inquiry" is a question that seeks information or a solution. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0023] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0024] First, the terms used in the following description will be explained.

[0025] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0026] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0027] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0028] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0029] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0030] [First embodiment]

[0031] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0032] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0033] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0034] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0035] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0036] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0037] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0038] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0039] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0040] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0041] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0042] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0043] An embodiment of the present invention will be described below as a specific example.

[0044] System Overview

[0045] This invention is a system that automates everything from reading invoices to determining whether they are qualified invoices, digitizing them, storing them, and responding to inquiries, in order to streamline invoice-related operations for businesses. This system works in cooperation with three parties: a server, a terminal, and a user.

[0046] System configuration

[0047] server

[0048] Receiving invoice data

[0049] Text extraction using OCR processing

[0050] Data analysis and item extraction using generative AI

[0051] Matching with the database of eligible bill issuers

[0052] Electronic data storage based on the judgment results

[0053] Responding to inquiries through the QA system

[0054] Terminal

[0055] Providing a user interface

[0056] Sending and receiving data to the server

[0057] User

[0058] Uploading an invoice

[0059] Checking the matching results

[0060] Inquiry for necessary information

[0061] Program processing explanation

[0062] Uploading and receiving invoices

[0063] Processing content

[0064] A user uploads an image or PDF file of the invoice to the system using a terminal.

[0065] The terminal sends the uploaded bill file to the server.

[0066] The server stores the received bill file in a temporary storage area.

[0067] Invoice scanning and OCR processing

[0068] Processing content

[0069] The server scans the received invoice file using OCR technology.

[0070] The server extracts text data from the invoice using OCR processing and stores it in a temporary database.

[0071] Data analysis and item extraction using generative AI

[0072] Processing content

[0073] The server uses the generative AI model to analyze and extract each item of the invoice (e.g., invoice number, issue date, amount, issuer information, etc.) from the OCR results.

[0074] The server stores the analyzed and extracted data in a database as structured data.

[0075] Verification of eligible invoice issuers

[0076] Processing content

[0077] The server accesses a database of eligible bill issuers and verifies the extracted issuer information.

[0078] Based on the result of the comparison, the server determines whether or not the invoice is a qualified invoice.

[0079] The server updates the result of the determination with the structured data.

[0080] Digitization and storage

[0081] Processing content

[0082] The server then generates the final invoice data as electronic data and stores it on a cloud platform.

[0083] The server notifies the terminal that the saving is complete.

[0084] QA Support

[0085] Processing content

[0086] The user makes a query using the terminal.

[0087] The terminal transmits the inquiry to the server.

[0088] The server uses generative AI to analyze the inquiry and generate an appropriate response.

[0089] The server sends the generated answer to the terminal and provides it to the user.

[0090] Specific examples

[0091] For example, when a business digitizes invoices and determines whether they are qualified invoices, the user uploads an image of the invoice to the system from their device. The server then extracts the invoice's text data using OCR processing, and uses generation AI to analyze and extract each item on the invoice. The extracted issuer information is compared with a database of qualified invoice issuers, and the invoice is determined to be qualified. Finally, the digitized invoice data is securely stored on a cloud platform, allowing users to access it at any time. Furthermore, if the user makes an inquiry, the generation AI can respond quickly.

[0092] The above is an embodiment of the present invention. This system automates invoice processing and significantly improves operational efficiency.

[0093] The processing flow will be explained below.

[0094] Step 1:

[0095] A user uploads an image or PDF file of the invoice to the system using a terminal.

[0096] Step 2:

[0097] The terminal stores the uploaded invoice file in a temporary storage area and transmits it to the server.

[0098] Step 3:

[0099] The server passes the received invoice file to the OCR processing engine and begins extracting character data.

[0100] Step 4:

[0101] The server retrieves the character data generated by the OCR processing engine and stores it in a temporary text database.

[0102] Step 5:

[0103] The server launches the generation AI, analyzes the saved text data, and extracts each item of the invoice (invoice number, issue date, amount, issuer information, etc.).

[0104] Step 6:

[0105] The server stores each extracted item in a database as structured data.

[0106] Step 7:

[0107] The server accesses a database of eligible bill issuers and verifies the extracted issuer information.

[0108] Step 8:

[0109] Based on the result of the check, the server determines whether the bill is a qualified bill.

[0110] Step 9:

[0111] The server stores the judgment results together with the structured data as electronic data on a cloud platform.

[0112] Step 10:

[0113] The server notifies the terminal that the saving has been completed, and the user is also notified.

[0114] Step 11:

[0115] When a user makes an inquiry using a terminal, the inquiry content is transmitted from the terminal to the server.

[0116] Step 12:

[0117] The server uses generative AI to analyze the inquiry and generate an appropriate response.

[0118] Step 13:

[0119] The server sends the generated answer to the terminal and provides it to the user.

[0120] Step 14:

[0121] The server performs daily backups of all data stored on the cloud platform to ensure data safety.

[0122] Step 15:

[0123] The server performs access control based on user privileges and maintains appropriate security for stored data.

[0124] The above is the specific processing flow in the system of the present invention, which automates the invoice processing work and significantly reduces the workload of business operators.

[0125] Example 1

[0126] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0127] Traditional invoice processing operations require a lot of manual input and verification, which takes time and effort. There is also a high risk of human error, making it necessary to streamline operations. Furthermore, delays in determining and responding to qualified invoices can expose businesses to legal risks and a decline in their credibility. Furthermore, the lack of a system for quickly responding to user inquiries makes it difficult to provide reliable services.

[0128] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0129] In this invention, the server includes means for receiving an invoice document file, means for converting the received file into character data using optical character recognition technology, means for analyzing the converted character data using a generation AI and extracting each item of the invoice, means for comparing the extracted invoice issuer information with a qualified invoice issuer database, means for determining whether the invoice is a qualified invoice based on the comparison results, means for storing the determined invoice as electronic data on a cloud platform, and means for generating responses to user inquiries using a generation AI. This reduces manual work, improves business efficiency and accuracy, and enables early determination of qualified invoices and secure electronic storage. Furthermore, user inquiries can be responded to quickly and accurately.

[0130] An "invoice document file" is a file that electronically stores a document used as an invoice, and refers to a digital file in image format, PDF format, or the like.

[0131] "Optical character recognition technology" is a technology that analyzes and extracts character information from images or documents containing handwritten characters, and is also known as OCR (Optical Character Recognition).

[0132] "Generative AI" is a technology that uses machine learning models and artificial intelligence to analyze and generate data, and is particularly used in natural language processing and image analysis.

[0133] The "Qualified Invoice Issuer Database" is a database for registering and managing information on businesses that issue qualified invoices, which allows you to check whether a specific business issues qualified invoices.

[0134] A "cloud platform" is a foundation for using infrastructure such as computing resources and storage provided via the Internet, and is provided as a cloud service.

[0135] "User inquiries" refer to requests made by system users to the system for information or to resolve questions.

[0136] This invention is a system designed to help businesses streamline invoice-related operations. This system functions through the cooperation of three parties: the server, the terminal, and the user. Specifically, it automates everything from uploading invoices, OCR processing, data analysis using AI generation, and determining whether an invoice is eligible, to digitizing and saving the data, and responding to inquiries.

[0137] System configuration

[0138] The system is divided into the following main components:

[0139] server

[0140] The server performs many functions, including:

[0141] 1. Receiving billing data: Receives the billing file sent from the user or terminal.

[0142] 2. OCR processing: Optical character recognition technology (e.g., Google® Cloud Vision API or Tesseract OCR) is used to extract character data from the received invoice file.

[0143] 3. Data analysis with generative AI: A generative AI model (e.g., OpenAI® GPT-4®) is used to parse and extract each invoice line item from the OCR results.

[0144] 4. Matching with the database of qualified invoice issuers: The extracted issuer information is matched with the database to determine whether the invoice is a qualified invoice.

[0145] 5. Electronic data conversion and storage: The adjudicated invoice data is electronically stored on a cloud platform (e.g., AWS (registered trademark) S3).

[0146] 6. Response to inquiries using a QA system: Using generative AI, we analyze user inquiries and generate appropriate responses.

[0147] Terminal

[0148] The terminal provides the following features:

[0149] 1. User Interface: Provides an interface for users to upload invoices, check matching results, and make inquiries.

[0150] 2. Data transmission and reception: Sends billing files and inquiry details to the server and receives responses from the server.

[0151] User

[0152] The user does the following:

[0153] 1. Upload invoice: Upload an image or PDF file of your invoice using our dedicated application or web interface.

[0154] 2. Check the matching results: Check the matching results generated by the server and the electronic invoice data.

[0155] 3. Inquiry: Submitting a request to the system for information or to resolve a question.

[0156] Specific examples

[0157] For example, if a business wants to digitize invoices and determine which invoices are eligible, it would follow the steps below:

[0158] 1. The user uses a terminal to upload an image of the invoice to the system. Specifically, typically, the user takes a photo of the invoice using the camera function of a smartphone and uploads the image file.

[0159] 2. The terminal sends the bill file to the server.

[0160] 3. The server receives the file and extracts text data from the invoice using OCR processing, using technologies such as Google Cloud Vision API and Tesseract OCR.

[0161] 4. The server uses a generative AI model (such as OpenAI GPT-4) to analyze and extract each item of the invoice (invoice number, issue date, amount, issuer information, etc.) from the OCR results.

[0162] 5. The server checks the issuer information against the database of qualified invoice issuers and determines that the invoice is a qualified invoice.

[0163] 6. The server electronically stores the adjudicated invoice data on a cloud platform (e.g., AWS S3).

[0164] 7. Users can access the stored data to check the matching results, and if they make inquiries, they can receive a prompt and appropriate response from the generating AI.

[0165] Prompt Sentence Examples

[0166] An example of an input to a generative AI model might be a prompt like this:

[0167] "A user has used their device to upload an image of an invoice. What are the next steps?"

[0168] By using this system, invoice processing operations can be automated, improving work efficiency and accuracy. Furthermore, it becomes possible to respond to user inquiries quickly and accurately.

[0169] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0170] Step 1: Upload your invoice

[0171] The user uploads an image or PDF file of the invoice to the system using the terminal. Specifically, the user clicks the "File Upload" button in the dedicated application or web interface, selects the relevant invoice file, and displays it.

[0172] Input: Invoice image or PDF file.

[0173] Output: The upload request sent to the device.

[0174] Step 2: Submit your invoice file

[0175] The terminal sends the invoice file uploaded by the user to the server using an HTTP POST request, along with the file metadata and user ID.

[0176] Input: User selected invoice file and metadata.

[0177] Output: The file sent to the server along with any accompanying information.

[0178] Step 3: Save the file temporarily

[0179] The server stores the received invoice file in a temporary storage area (for example, an Amazon S3 bucket or local storage). The file name includes the user ID and a timestamp to make it uniquely identifiable.

[0180] Input: Invoice file sent from the terminal.

[0181] Output: Temporarily saved file.

[0182] Step 4: OCR processing

[0183] The server scans the invoice file using OCR technology (Google Cloud Vision API and Tesseract OCR) and extracts text data from the image. After OCR processing, the extracted text data is stored in a temporary database.

[0184] Input: Temporarily saved invoice file.

[0185] Output: The extracted text data.

[0186] Step 5: Temporarily save the text data

[0187] The server stores the text data obtained through the OCR process in a temporary database, and checks the data integrity in the process, specifically logging any incomplete or erroneous data in case it needs to be reprocessed.

[0188] Input: Text data extracted by OCR processing.

[0189] Output: Temporarily saved text data and processing log.

[0190] Step 6: Data analysis and item extraction using generative AI

[0191] The server uses a generative AI model (such as OpenAI GPT-4) to parse and extract each item of the invoice from the OCR results (invoice number, issue date, amount, issuer information, etc.), specifically identifying important entities and organizing them into appropriate fields in a database.

[0192] Input: Temporarily saved text data.

[0193] Output: Structured item data.

[0194] Step 7: Save the analysis data

[0195] The server stores the analyzed and extracted data in this database as structured data in JSON or SQL table format to facilitate subsequent query processing.

[0196] Input: Item data analyzed and extracted by the generative AI.

[0197] Output: Structured data stored in this database.

[0198] Step 8: Match eligible invoice issuers

[0199] The server accesses a database of eligible bill issuers and matches the extracted issuer information by searching the issuer information using a query to determine whether there is a match.

[0200] Input: Structured issuer information.

[0201] Output: Issuer match result.

[0202] Step 9: Update the results

[0203] Based on the result of the matching, the server determines whether the invoice is a qualified invoice or not, and updates the result along with the structured data, adding a determination flag and status information.

[0204] Input: Issuer match result.

[0205] Output: Updated decision result data.

[0206] Step 10: Digitization and storage

[0207] The server generates the final invoice data as electronic data and stores it securely on a cloud platform (such as AWS S3). The data is also encrypted when stored.

[0208] Input: Judgment result data.

[0209] Output: Electronic data stored on a cloud platform.

[0210] Step 11: Sending a save completion notification

[0211] The server generates a notification of completion of saving and sends it to the terminal. The notification includes the URL of the invoice data and status information.

[0212] Input: Saved invoice data.

[0213] Output: Notifications sent to the device.

[0214] Step 12: Submitting an inquiry

[0215] A user uses a terminal to make an inquiry to the system. Specifically, the user enters the inquiry details in the input form and clicks the send button.

[0216] Input: Enquiry details.

[0217] Output: The query request generated on the terminal.

[0218] Step 13: Sending the query request to the server

[0219] The terminal converts the user's inquiry into JSON format and sends it to the server via an HTTP POST request.

[0220] Input: User's inquiry.

[0221] Output: The request sent to the server.

[0222] Step 14: Generate and send a response using the generative AI

[0223] The server uses generative AI (such as OpenAI GPT-4) to analyze the query and generate an appropriate answer, which is then sent to the device and served to the user.

[0224] Input: User's inquiry.

[0225] Output: The response sent to the terminal.

[0226] This concludes the detailed explanation of each processing step of the system. By implementing this system, invoice processing operations can be automated and made more efficient.

[0227] (Application example 1)

[0228] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0229] In conventional logistics centers, product inspection work is done manually, which is inefficient and prone to human error. Furthermore, inspectors need a great deal of prior knowledge and experience to determine whether a product is suitable. Furthermore, responding to inquiries is also manual, which often results in a long response time. There is a need to solve these problems and improve the operational efficiency of logistics centers.

[0230] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0231] In this invention, the server includes a means for receiving an image file of an invoice or label, a means for converting the received file into character data using OCR technology, and a means for analyzing the converted character data using a generation AI to extract each item. This enables the automation of product inspection work at logistics centers and the determination of qualified products.

[0232] "Invoice or label image file" means a scanned image or digital file saved in PDF format of a label or invoice for a product handled at a distribution center or other operation.

[0233] "OCR technology" is an abbreviation for optical character recognition technology, which extracts character information such as letters and numbers from image files and converts them into digital data.

[0234] "Generative AI" is a type of artificial intelligence that uses machine learning algorithms to analyze large amounts of data and generate new information and answers.

[0235] The "eligibility determination database" is a database for collating extracted issuer information or product information to determine its eligibility.

[0236] "Cloud infrastructure" refers to cloud computing infrastructure provided via the Internet, and is a platform for storing, processing, and managing data.

[0237] An "enquiry" is a question or question posed by a user to the system for which the system will generate an answer.

[0238] System Overview

[0239] This invention is a system for automating product inspection work at logistics centers. The system uses smart glasses to scan product labels, analyzes the information in real time, determines whether the product is eligible, and responds to inquiries as needed. The information is then securely stored on a cloud platform.

[0240] System configuration

[0241] Smart Glasses

[0242] The product label is scanned and the image data is sent to the server.

[0243] server

[0244] The received image data is converted into character data using OCR technology.

[0245] The generated text data is analyzed using generative AI to extract each item.

[0246] The extracted product information is compared with the eligibility assessment database, and the assessment results are stored on a cloud platform.

[0247] Generative AI is used to generate responses to user inquiries.

[0248] Cloud infrastructure

[0249] The determined data is stored securely and provided so that the user can access it.

[0250] Specific implementation steps

[0251] 1. Initiate inspection and scan:

[0252] A user scans a product label using smart glasses.

[0253] The smart glasses send the scanned image data to a server.

[0254] 2. Data extraction using OCR processing:

[0255] The server converts the received image data into text data using OCR technology (for example, Google Cloud Vision API).

[0256] The converted character data is temporarily saved.

[0257] 3. Data analysis and eligibility judgment by generative AI:

[0258] The server uses a generative AI model (e.g., OpenAI GPT model) to analyze the text data and extract each item, such as product code, quantity, and expiration date.

[0259] The extracted product information is compared with an eligibility determination database to determine the product's eligibility.

[0260] The determined data is stored on a cloud platform (e.g., AWS S3).

[0261] 4. QA Support:

[0262] When a user makes a query, the server uses generative AI to analyze the query and generate an appropriate answer.

[0263] The server provides the generated answer to the user.

[0264] Specific examples

[0265] Warehouse workers at the distribution center use smart glasses to scan the labels on incoming products. The server analyzes the data in real time and immediately determines whether the product is eligible. If it is eligible, the evidence is stored in AWS S3. If it is not eligible, the system notifies the user of the appropriate measures. Inquiries are also handled promptly using the generative AI model.

[0266] Example prompts to input to a generative AI model:

[0267] "Analyze the information on the product label scanned by the smart glasses, extract data such as the product code, quantity, and expiration date, and then use that information to determine whether the product is eligible. Also, generate an appropriate response to inquiries from users. If the product is eligible, store the data in a cloud platform and notify the user."

[0268] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0269] Step 1:

[0270] A user scans a product label using smart glasses.

[0271] Input: A user puts on the smart glasses and captures an image of a product label.

[0272] Output: The captured image data.

[0273] What it does: When the smart glasses see a product label, they activate their camera and scan the image of the label.

[0274] Step 2:

[0275] The smart glasses send the scanned image data to a server.

[0276] Input: Scanned product label image.

[0277] Output: Image data sent to the server.

[0278] How it works: The smart glasses upload the label image data to the server via a wireless network.

[0279] Step 3:

[0280] The server converts the received image data into text data using OCR technology.

[0281] Input: Received label image data.

[0282] Output: Character data extracted by OCR technology.

[0283] What it does: The server uses an OCR service such as Google Cloud Vision API to extract text information from the label image.

[0284] Step 4:

[0285] The server uses a generative AI model to analyze the converted text data and extract each item.

[0286] Input: Character data obtained by OCR processing.

[0287] Output: Information about each item, such as product code, quantity, and expiration date.

[0288] How it works: The server uses the OpenAI GPT model to analyze the text data and identify and extract the required data items.

[0289] Step 5:

[0290] The server compares the extracted product information with an eligibility determination database to determine eligibility.

[0291] Input: Extracted product information (product code, quantity, expiration date, etc.).

[0292] Output: Product eligibility determination result.

[0293] What happens: The server checks the product information against the eligibility database to determine if the product meets the criteria.

[0294] Step 6:

[0295] The server stores the determined data on a cloud platform.

[0296] Input: Data for the determined eligible product.

[0297] Output: Eligible data stored in a cloud infrastructure.

[0298] What it does: The server stores the data in cloud storage such as AWS S3, making it available for users to access later.

[0299] Step 7:

[0300] The server uses a generation AI to generate responses to user inquiries.

[0301] Input: User's inquiry.

[0302] Output: The generated response to the query.

[0303] Specific operation: Using a generative AI model (e.g., OpenAI GPT model), the query content is analyzed, an optimal answer is generated, and provided to the user.

[0304] In this way, by carrying out specific processing at each step, product inspection work at the logistics center is automated efficiently and accurately.

[0305] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0306] An embodiment of the present invention will be described below as a specific example.

[0307] System Overview

[0308] This invention is a system that allows businesses to improve the efficiency of their invoice-related operations by scanning invoices, determining whether they are qualified, digitizing them, storing them, responding to inquiries, and recognizing user emotions to optimize responses. This system works in cooperation with the server, terminals, and users, and makes use of a variety of technologies.

[0309] System configuration

[0310] server

[0311] Receiving invoice data

[0312] Text extraction using OCR processing

[0313] Data analysis and item extraction using generative AI

[0314] Matching with the database of eligible bill issuers

[0315] Electronic data storage based on the judgment results

[0316] Responding to inquiries through the QA system

[0317] Emotional analysis of users using an emotion engine

[0318] Response optimization based on emotional data

[0319] Learning database updates

[0320] Terminal

[0321] Providing a user interface

[0322] Sending and receiving data to the server

[0323] User

[0324] Uploading an invoice

[0325] Checking the matching results

[0326] Inquiry for necessary information

[0327] Program processing explanation

[0328] Uploading and receiving invoices

[0329] Processing content

[0330] A user uploads an image or PDF file of the invoice to the system using a terminal.

[0331] The terminal stores the uploaded invoice file in a temporary storage area and transmits it to the server.

[0332] The server stores the received bill file in a temporary storage area.

[0333] Invoice scanning and OCR processing

[0334] Processing content

[0335] The server scans the received invoice file using OCR technology.

[0336] The server extracts text data from the invoice using OCR processing and stores it in a temporary database.

[0337] Data analysis and item extraction using generative AI

[0338] Processing content

[0339] The server uses the generative AI model to analyze and extract each item of the invoice (e.g., invoice number, issue date, amount, issuer information, etc.) from the OCR results.

[0340] The server stores the analyzed and extracted data in a database as structured data.

[0341] Verification of eligible invoice issuers

[0342] Processing content

[0343] The server accesses a database of eligible bill issuers and verifies the extracted issuer information.

[0344] Based on the result of the comparison, the server determines whether or not the invoice is a qualified invoice.

[0345] The server updates the result of the determination with the structured data.

[0346] Digitization and storage

[0347] Processing content

[0348] The server then generates the final invoice data as electronic data and stores it on a cloud platform.

[0349] The server notifies the terminal that the saving is complete.

[0350] QA Support and Emotion Engine

[0351] Processing content

[0352] The user makes a query using the terminal.

[0353] The terminal transmits the inquiry to the server.

[0354] The server uses generative AI to analyze the inquiry and generate an appropriate response.

[0355] The server uses an emotion engine as part of the QA system to analyze the user's emotions.

[0356] The server adjusts the content and tone of the response based on the analyzed emotional data to generate the optimal response.

[0357] The server sends the generated answer to the terminal and provides it to the user.

[0358] The server stores the emotional data in a learning database and uses it to improve the response accuracy of the generation AI.

[0359] Specific examples

[0360] For example, when a business digitizes invoices and determines whether they are qualified invoices, the user uploads an image of the invoice to the system from their device. The server then extracts the invoice's text data using OCR processing and uses generation AI to analyze and extract each item on the invoice. The extracted issuer information is compared with a database of qualified invoice issuers, and the invoice is determined to be qualified. Finally, the digitized invoice data is securely stored on a cloud platform, allowing users to access it at any time.

[0361] Additionally, when a user makes an inquiry, the server uses generative AI and an emotion engine to analyze the user's emotions and generate an appropriate response. For example, if the user is feeling anxious or angry, providing a polite response that reflects those emotions can improve user satisfaction. At the same time, this emotional data is stored in a learning database and used to improve the accuracy of future response generation.

[0362] The above is an embodiment of the present invention. This system not only automates bill processing but also improves the user experience.

[0363] The processing flow will be explained below.

[0364] Step 1:

[0365] A user uploads an image or PDF file of the invoice to the system using a terminal.

[0366] Step 2:

[0367] The terminal stores the uploaded invoice file in a temporary storage area and transmits it to the server.

[0368] Step 3:

[0369] The server passes the received invoice file to the OCR processing engine and begins extracting character data.

[0370] Step 4:

[0371] The server retrieves the character data generated by the OCR processing engine and stores it in a temporary text database.

[0372] Step 5:

[0373] The server launches the generation AI, analyzes the saved text data, and extracts each item of the invoice (invoice number, issue date, amount, issuer information, etc.).

[0374] Step 6:

[0375] The server stores each extracted item in a database as structured data.

[0376] Step 7:

[0377] The server accesses a database of eligible bill issuers and verifies the extracted issuer information.

[0378] Step 8:

[0379] Based on the result of the check, the server determines whether the bill is a qualified bill.

[0380] Step 9:

[0381] The server stores the judgment results together with the structured data as electronic data on a cloud platform.

[0382] Step 10:

[0383] The server notifies the terminal that the saving has been completed, and the user is also notified.

[0384] Step 11:

[0385] When a user makes an inquiry using a terminal, the inquiry content is transmitted from the terminal to the server.

[0386] Step 12:

[0387] The server uses generative AI to analyze the inquiry and generate an appropriate response.

[0388] Step 13:

[0389] The server uses an emotion engine as part of the QA system to analyze the user's emotions.

[0390] Step 14:

[0391] The server adjusts the content and tone of the response based on the analyzed emotional data to generate the optimal response.

[0392] Step 15:

[0393] The server sends the generated answer to the terminal and provides it to the user.

[0394] Step 16:

[0395] The server stores the emotional data in a learning database and uses it to improve the response accuracy of the generation AI.

[0396] Step 17:

[0397] The server performs daily backups of all data stored on the cloud platform to ensure data safety.

[0398] Step 18:

[0399] The server performs access control based on user privileges and maintains appropriate security for stored data.

[0400] The above is the specific processing flow in the system of the present invention, which automates invoice processing, significantly reducing the workload of business operators and improving the user experience.

[0401] Example 2

[0402] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0403] Conventional invoice processing systems have the problem of requiring a great deal of manual work and time for scanning, analyzing, judging, digitally storing invoices, and responding to user inquiries. Furthermore, they can only provide a uniform response to user inquiries, making it difficult to improve user satisfaction. To solve these problems, there is a need for a system that can fully automate invoice processing and generate appropriate responses based on the user's emotions.

[0404] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0405] In this invention, the server includes means for receiving an image or electronic file of an invoice, means for converting the received file into character data using optical character recognition technology, means for analyzing the converted character data using a generative AI model and extracting each item of the invoice, means for comparing the extracted invoice issuer information with a database of qualified invoice issuers, means for determining whether the invoice is a qualified invoice based on the comparison results, means for storing the determined invoice as electronic data on a cloud platform, means for generating responses to inquiries from users using the generative AI model, means for using an emotion analysis engine to analyze user emotions and adjust the content and tone of the response, and means for saving the emotion analysis data in a learning database and improving the response accuracy of the generative AI model. This enables full automation of invoice processing operations and the generation of appropriate responses according to user emotions.

[0406] 1. "Invoice" means a document used to request payment for goods or services.

[0407] 2. "Image" means any form in which visual data can be stored and displayed electronically.

[0408] 3. "Electronic files" refers to documents and data stored and managed in digital form.

[0409] 4. "Receiving" refers to obtaining data or information from an external source.

[0410] 5. "Optical character recognition technology" is a technology that analyzes letters and numbers in an image and converts them into text data.

[0411] 6. “Generative AI model” means a model that uses artificial intelligence to generate and analyze data.

[0412] 7. "Analysis" refers to examining data in detail to clarify its content and structure.

[0413] 8. "Item" means an individual element that refers to a specific piece of information or data.

[0414] 9. "Qualified Bill Issuer Database" is a database for verifying the eligibility of bill issuers.

[0415] 10. "Matching" refers to comparing different data to determine whether they match or fit.

[0416] 11. "Judgment" means determining suitability based on criteria.

[0417] 12. "Electronic data" means data stored and managed in digital form.

[0418] 13. "Cloud infrastructure" refers to infrastructure with data storage and processing capabilities provided via the Internet.

[0419] 14. "Inquiry" means a question or request for information or an answer.

[0420] 15. "Response" means an answer or reaction provided to an inquiry or request.

[0421] 16. "Sentiment analysis engine" refers to software technology for analyzing emotions from data such as text and voice.

[0422] 17. "Training Database" means a database that stores and manages data used to train machine learning models.

[0423] This invention is a system for streamlining invoice-related tasks, and it works in cooperation with the server, terminal, and user. Specifically, it is equipped with technology for scanning invoices, determining whether they are qualified invoices, digitizing them, storing them, responding to inquiries, and recognizing user emotions to optimize responses.

[0424] System configuration

[0425] Hardware and Software Configuration

[0426] Server: Use a high-performance cloud server (e.g., AWS EC2).

[0427] Device: The computer or smartphone used by the user.

[0428] OCR technology: Uses Google Cloud Vision API.

[0429] Generative AI model: OpenAI GPT-4 is used.

[0430] Cloud infrastructure: AWS S3 is used for data storage.

[0431] Sentiment analysis engine: Uses Microsoft (registered trademark) Azure (registered trademark) Emotion API.

[0432] Specific explanation of program processing

[0433] Uploading and receiving invoices

[0434] Users upload invoice images or PDF files to the system using a dedicated web interface on their device or a smartphone application. The device temporarily stores the uploaded file in local storage and then sends it to the server using HTTPS. The server then stores the file in a temporary storage area (AWS S3 bucket).

[0435] Invoice scanning and OCR processing

[0436] The server scans the received invoice file using Google Cloud Vision API and extracts the text data, which is then stored in a temporary database (MongoDB).

[0437] Data analysis and item extraction using generative AI

[0438] The server uses OpenAI GPT-4 to analyze and extract each item of the invoice (e.g., invoice number, issue date, amount, issuer information, etc.) from the OCR processing results. The analysis is performed by inputting the following prompt sentence into the generation AI:

[0439] Extract the invoice items from this text.

[0440] The analyzed and extracted data is stored in a database (MySQL (registered trademark)) as structured data (e.g., database records).

[0441] Verification of eligible invoice issuers

[0442] The server accesses the database of qualified invoice issuers (a database of the National Tax Agency) and compares the issuer information extracted by the generation AI. Based on the comparison results, the server determines that the invoice is a qualified invoice and updates the internal database.

[0443] Digitization and storage

[0444] The server generates the final invoice data as electronic data (e.g. PDF format) and saves it on the cloud infrastructure (AWS S3). It notifies the terminal that saving is complete.

[0445] QA Support and Emotion Engine

[0446] The user enters a question into a dedicated inquiry form and presses the send button. The device sends the information to the server, which uses OpenAI GPT-4 to analyze the inquiry and generate an appropriate answer. The server also analyzes the user's emotions using the Microsoft Azure Emotion API and adjusts the content and tone of the response. For example, the following prompt sentences can be used:

[0447] I have a question regarding invoice number 12345. The invoice was determined to be ineligible. I would like to know the reason for this and how I can resubmit it as an eligible invoice.

[0448] The final answer generated is provided to the user, and the sentiment analysis data is stored in a learning database to help improve the accuracy of future response generation.

[0449] Specific examples

[0450] For example, when a business digitizes invoices and determines whether they are qualified invoices, the user uploads an image of the invoice to the system from their device. The server uses OCR processing to extract the text data from the invoice, and uses generation AI to analyze and extract each item on the invoice. The extracted issuer information is compared with a database of qualified invoice issuers, and the invoice is determined to be qualified. The digitized invoice data is securely stored on a cloud platform, allowing users to access it at any time.

[0451] Additionally, when a user makes an inquiry, the server uses generative AI and an emotion engine to analyze the user's emotions and generate an appropriate response. For example, if a user is feeling anxious or angry, providing a polite response that reflects those emotions can improve user satisfaction. At the same time, this emotional data is stored in a learning database and used to improve the accuracy of future response generation. This system significantly improves the efficiency of invoice processing and the user experience.

[0452] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0453] Step 1: Upload your invoice

[0454] Users upload invoice images or PDF files to the system using a dedicated web interface or smartphone app. The input is the invoice image file or PDF file, and the output is saving it to the terminal's temporary storage area. The terminal temporarily saves the uploaded file in local storage and sends it to the server using the HTTPS protocol.

[0455] Step 2: Receiving and temporarily saving the file

[0456] The server receives the invoice file sent from the terminal and saves it in a temporary storage area within the cloud infrastructure (e.g. AWS S3). The input is the invoice file sent from the terminal, and the output is the file saved in the temporary storage area within the cloud infrastructure. The file is checked for integrity and scanned for viruses, and if there are no problems, it proceeds to the next processing step.

[0457] Step 3: OCR the invoice

[0458] The server scans the invoice file stored in the temporary storage area of ​​the cloud infrastructure using the Google Cloud Vision API. The input is the invoice file stored in the temporary storage area, and the output is text data. Specifically, the letters and numbers in the image are extracted as text data, and the data is stored in a temporary database (e.g., MongoDB).

[0459] Step 4: Data analysis with generative AI

[0460] The server uses OpenAI GPT-4 to analyze and extract each item of the invoice based on the extracted character data. The input is the extracted character data, and the output is structured data including each item of the invoice (e.g., invoice number, issue date, amount, issuer information, etc.). The following prompt sentence is input to the generation AI:

[0461] Extract the invoice items from this text.

[0462] The generated structured data is stored in a database (e.g., MySQL).

[0463] Step 5: Verify eligible billing entities

[0464] The server compares the issuer information stored in the database with the database of qualified invoice issuers (a database of the National Tax Agency). The input is the issuer information, and the output is the eligibility determination result. Specifically, the server compares the issuer information with the database and determines whether the invoice is appropriate based on the results.

[0465] Step 6: Save the relevance data

[0466] The server updates the database with structured data that reflects the judgment result. The input is the eligibility judgment result, and the output is the updated structured data.

[0467] Step 7: Electronic data generation and final storage

[0468] The server generates the final invoice data that has been determined to be eligible as electronic data and stores it on a cloud platform (e.g., AWS S3). The input is the determined invoice data, and the output is the final electronic data. A notification that the data has been saved is then sent to the terminal.

[0469] Step 8: Receiving inquiries from users

[0470] The user enters a question into a dedicated inquiry form and presses the send button. The input is the inquiry content, and the output is the transmission of the inquiry content to the server. The terminal then sends the inquiry content to the server.

[0471] Step 9: Parsing the query and generating a response

[0472] The server analyzes the query using OpenAI GPT-4 and generates an appropriate answer. The input is the query, and the output is the generated answer. It also analyzes the user's emotions using Microsoft Azure Emotion API and adjusts the content and tone of the response. For example, it uses the following prompt:

[0473] I have a question regarding invoice number 12345. The invoice was determined to be ineligible. I would like to know the reason for this and how I can resubmit it as an eligible invoice.

[0474] Step 10: Storing emotion data and serving responses

[0475] The server stores the generated answer together with the emotion analysis data in a learning database and transmits the answer to the terminal. The input is the generated answer and the emotion analysis data, and the output is saving the answer in the learning database and providing the answer to the user.

[0476] The above is the specific flow of operations in the system's processing steps. This system will greatly improve the efficiency of invoice processing and the user experience.

[0477] (Application example 2)

[0478] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0479] Improving customer service efficiency and customer satisfaction are currently key challenges for many brick-and-mortar stores. In particular, quick and accurate responses are required when handling inquiries and troubleshooting, which directly impacts the customer experience. However, handling a large number of inquiries requires human intervention, making it difficult to secure and manage a workforce. Furthermore, while properly understanding customer emotions and responding accordingly could contribute to further improving customer satisfaction, automation remains lacking in this area.

[0480] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for receiving an image or electronic document of an invoice; means for converting the received document into character data using optical character recognition technology; means for analyzing the converted character data using generative artificial intelligence and extracting each item of the invoice; means for comparing the extracted invoice issuer information with a database of qualified invoice issuers; means for determining whether the invoice is a qualified invoice based on the comparison result; means for storing the determined invoice as electronic data on a remote platform; means for generating a response to a user inquiry using generative artificial intelligence; and means for adjusting the response based on the emotion data using an emotion recognition engine that analyzes the user's emotions. This makes it possible for brick-and-mortar stores to automatically provide optimal responses based on the analysis of receipt and invoice information and the customer's emotions.

[0481] "Invoice image or electronic document" refers to an image file or a digital document such as a PDF that visually represents the contents of the invoice.

[0482] "Optical character recognition technology" is a technology for analyzing character information contained in an image and converting it into text data.

[0483] "Generative artificial intelligence" is an artificial intelligence technology that generates new data and responses based on large amounts of data, and is used for natural language generation and various data analyses.

[0484] The "Qualified Invoice Issuer Database" is a database that compiles information on businesses that have the authority to issue qualified invoices.

[0485] "Remote infrastructure" refers to cloud environments and remote data storage accessible via the Internet.

[0486] An "emotion recognition engine" is an engine that analyzes and identifies a user's emotional state from data such as text and voice.

[0487] "User emotion" refers to the user's psychological state and emotional expression during an inquiry or conversation.

[0488] "Response" or "response generation" is the process of automatically creating an appropriate reply to a user's inquiry using generative artificial intelligence.

[0489] System Overview

[0490] The system of the present invention includes a series of processes, from reading invoices to determining whether they are eligible, converting them into electronic data, storing them, responding to inquiries, and recognizing the user's emotions to generate the most appropriate response. This system works in cooperation with the server, terminals, and users, and makes full use of a variety of technologies, including OCR technology, generative artificial intelligence, and an emotion recognition engine.

[0491] Server Features

[0492] The server has the following main functions:

[0493] 1. Receiving invoices: Receiving invoice images and electronic documents uploaded by users.

[0494] 2. Optical character recognition processing: The received document is converted into character data using optical character recognition technology (such as pytesseract).

[0495] 3. Data analysis and item extraction: Using generative artificial intelligence (such as Hugging Face's transformers library), each item of the invoice (e.g., invoice number, issue date, amount, issuer information, etc.) is analyzed and extracted from the converted text data.

[0496] 4. Database matching: The extracted invoice issuer information is matched with the database of eligible invoice issuers to determine whether the invoice is eligible.

[0497] 5. Digitalization and storage: The judged invoices are stored as electronic data on a cloud platform.

[0498] 6. Inquiry handling and response generation: Generative artificial intelligence is used to generate appropriate responses to user inquiries.

[0499] 7. Sentiment Analysis: An emotion recognition engine is used to analyze the user's emotional data and tailor the response.

[0500] Device Features

[0501] The terminal provides the following functions to the user:

[0502] 1. Provide an interface: Through the user interface, you can upload invoices, send and receive inquiries, etc.

[0503] 2. Sending and receiving data: Two-way communication with the server to send or receive data.

[0504] User operations

[0505] The user does the following:

[0506] 1. Uploading invoices: Use the terminal to upload an image or electronic document of the invoice to the system.

[0507] 2. Sending a query: Use the terminal to query the system for the required information.

[0508] 3. Check the matching result: Check the matching result sent from the server.

[0509] Hardware and software used

[0510] The hardware used is a customer-facing robot from a brick-and-mortar store (e.g., a typical customer-facing robot), while the software uses OCR technology (pytesseract), generative artificial intelligence (Hugging Face's transformers library and GPT-2 model), and a sentiment analysis module (sentiment_analysis).

[0511] Specific examples

[0512] For example, if a customer asks "Can I get a refund for this item?" in a store, the robot will analyze the customer's facial expressions and tone of voice to determine their emotions, and if the emotion score is low, it will generate a more polite and flexible response. At the same time, if the customer shows the robot a receipt, it will use OCR technology to quickly read the information on the receipt and immediately determine whether a refund is possible and respond accordingly.

[0513] Prompt Sentence Examples

[0514] User Question: "Can I get a refund for this item?"

[0515] Depending on the sentiment analysis results, the generated response may be:

[0516] "Of course, I'm happy to refund this item." (High sentiment score)

[0517] "Sorry, but we need to go through some checks before we can process your refund. Could you please wait a moment?" (if your sentiment score is low)

[0518] This system is expected to improve the efficiency of customer service in physical stores and increase customer satisfaction.

[0519] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0520] Step 1:

[0521] The server receives the invoice image or electronic document uploaded by the user. The input format can be an image file, a PDF file, or other format. The received file is stored in a temporary storage area.

[0522] Step 2:

[0523] The server converts the received invoice into text data using optical character recognition (OCR). It scans the image or PDF file of the invoice received as input and generates text data as output. Specifically, it uses pytesseract to analyze the characters in the image or PDF and convert them into text format.

[0524] Step 3:

[0525] The server uses generative artificial intelligence to analyze and extract each item of the invoice (e.g., invoice number, issue date, amount, issuer information, etc.) from the converted character data. It uses text data generated by OCR as input and generates structured data containing each item of the invoice as output. Specifically, it uses a generative artificial intelligence model using Hugging Face's transformers library.

[0526] Step 4:

[0527] The server compares the extracted invoice issuer information with the database of qualified invoice issuers. It uses the issuer information and database information as input and obtains the matching result as output. This completes the process of determining whether the invoice is qualified.

[0528] Step 5:

[0529] Based on the judgment results, the server stores qualified invoices as electronic data on a cloud platform. Using the matching results and invoice data as input, it generates a message indicating that the data has been saved as output. The electronic data is securely stored in cloud storage and can be accessed by users.

[0530] Step 6:

[0531] When a user makes a query using a terminal, the server receives the query content. At this time, the query content is sent as input and stored on the server side for processing.

[0532] Step 7:

[0533] The server uses generative artificial intelligence to generate appropriate responses to user queries. It uses the user's query as input and obtains the generated response text as output. Specifically, it uses Hugging Face's transformers library to generate the optimal response corresponding to the query.

[0534] Step 8:

[0535] The server uses an emotion recognition engine to analyze the user's emotion data. It uses the query content, the user's tone of voice, and text data as input, and generates an emotion score as output. It uses a sentiment analysis module (such as Sentiment Analysis).

[0536] Step 9:

[0537] The server adjusts the content and tone of the response based on the analyzed emotion data. It uses the emotion score and the initial response text as input and generates an optimized response text as output, which provides an appropriate response that corresponds to the user's emotion.

[0538] Step 10:

[0539] The server sends the generated response to the terminal and provides it to the user, using the optimized response text as input and displaying it on the terminal as output, after which the user can check the generated response.

[0540] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0541] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0542] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0543] [Second embodiment]

[0544] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0545] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0546] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0547] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0548] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0549] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0550] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0551] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0552] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0553] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0554] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0555] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0556] An embodiment of the present invention will be described below as a specific example.

[0557] System Overview

[0558] This invention is a system that automates everything from reading invoices to determining whether they are qualified invoices, digitizing them, storing them, and responding to inquiries, in order to streamline invoice-related operations for businesses. This system works in cooperation with three parties: a server, a terminal, and a user.

[0559] System configuration

[0560] server

[0561] Receiving invoice data

[0562] Text extraction using OCR processing

[0563] Data analysis and item extraction using generative AI

[0564] Matching with the database of eligible bill issuers

[0565] Electronic data storage based on the judgment results

[0566] Responding to inquiries through the QA system

[0567] Terminal

[0568] Providing a user interface

[0569] Sending and receiving data to the server

[0570] User

[0571] Uploading an invoice

[0572] Checking the matching results

[0573] Inquiry for necessary information

[0574] Program processing explanation

[0575] Uploading and receiving invoices

[0576] Processing content

[0577] A user uploads an image or PDF file of the invoice to the system using a terminal.

[0578] The terminal sends the uploaded bill file to the server.

[0579] The server stores the received bill file in a temporary storage area.

[0580] Invoice scanning and OCR processing

[0581] Processing content

[0582] The server scans the received invoice file using OCR technology.

[0583] The server extracts text data from the invoice using OCR processing and stores it in a temporary database.

[0584] Data analysis and item extraction using generative AI

[0585] Processing content

[0586] The server uses the generative AI model to analyze and extract each item of the invoice (e.g., invoice number, issue date, amount, issuer information, etc.) from the OCR results.

[0587] The server stores the analyzed and extracted data in a database as structured data.

[0588] Verification of eligible invoice issuers

[0589] Processing content

[0590] The server accesses a database of eligible bill issuers and verifies the extracted issuer information.

[0591] Based on the result of the comparison, the server determines whether or not the invoice is a qualified invoice.

[0592] The server updates the result of the determination with the structured data.

[0593] Digitization and storage

[0594] Processing content

[0595] The server then generates the final invoice data as electronic data and stores it on a cloud platform.

[0596] The server notifies the terminal that the saving is complete.

[0597] QA Support

[0598] Processing content

[0599] The user makes a query using the terminal.

[0600] The terminal transmits the inquiry to the server.

[0601] The server uses generative AI to analyze the inquiry and generate an appropriate response.

[0602] The server sends the generated answer to the terminal and provides it to the user.

[0603] Specific examples

[0604] For example, when a business digitizes invoices and determines whether they are qualified invoices, the user uploads an image of the invoice to the system from their device. The server then extracts the invoice's text data using OCR processing, and uses generation AI to analyze and extract each item on the invoice. The extracted issuer information is compared with a database of qualified invoice issuers, and the invoice is determined to be qualified. Finally, the digitized invoice data is securely stored on a cloud platform, allowing users to access it at any time. Furthermore, if the user makes an inquiry, the generation AI can respond quickly.

[0605] The above is an embodiment of the present invention. This system automates invoice processing and significantly improves operational efficiency.

[0606] The processing flow will be explained below.

[0607] Step 1:

[0608] A user uploads an image or PDF file of the invoice to the system using a terminal.

[0609] Step 2:

[0610] The terminal stores the uploaded invoice file in a temporary storage area and transmits it to the server.

[0611] Step 3:

[0612] The server passes the received invoice file to the OCR processing engine and begins extracting character data.

[0613] Step 4:

[0614] The server retrieves the character data generated by the OCR processing engine and stores it in a temporary text database.

[0615] Step 5:

[0616] The server launches the generation AI, analyzes the saved text data, and extracts each item of the invoice (invoice number, issue date, amount, issuer information, etc.).

[0617] Step 6:

[0618] The server stores each extracted item in a database as structured data.

[0619] Step 7:

[0620] The server accesses a database of eligible bill issuers and verifies the extracted issuer information.

[0621] Step 8:

[0622] Based on the result of the check, the server determines whether the bill is a qualified bill.

[0623] Step 9:

[0624] The server stores the judgment results together with the structured data as electronic data on a cloud platform.

[0625] Step 10:

[0626] The server notifies the terminal that the saving has been completed, and the user is also notified.

[0627] Step 11:

[0628] When a user makes an inquiry using a terminal, the inquiry content is transmitted from the terminal to the server.

[0629] Step 12:

[0630] The server uses generative AI to analyze the inquiry and generate an appropriate response.

[0631] Step 13:

[0632] The server sends the generated answer to the terminal and provides it to the user.

[0633] Step 14:

[0634] The server performs daily backups of all data stored on the cloud platform to ensure data safety.

[0635] Step 15:

[0636] The server performs access control based on user privileges and maintains appropriate security for stored data.

[0637] The above is the specific processing flow in the system of the present invention, which automates the invoice processing work and significantly reduces the workload of business operators.

[0638] Example 1

[0639] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0640] Traditional invoice processing operations require a lot of manual input and verification, which takes time and effort. There is also a high risk of human error, making it necessary to streamline operations. Furthermore, delays in determining and responding to qualified invoices can expose businesses to legal risks and a decline in their credibility. Furthermore, the lack of a system for quickly responding to user inquiries makes it difficult to provide reliable services.

[0641] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0642] In this invention, the server includes means for receiving an invoice document file, means for converting the received file into character data using optical character recognition technology, means for analyzing the converted character data using a generation AI and extracting each item of the invoice, means for comparing the extracted invoice issuer information with a qualified invoice issuer database, means for determining whether the invoice is a qualified invoice based on the comparison results, means for storing the determined invoice as electronic data on a cloud platform, and means for generating responses to user inquiries using a generation AI. This reduces manual work, improves business efficiency and accuracy, and enables early determination of qualified invoices and secure electronic storage. Furthermore, user inquiries can be responded to quickly and accurately.

[0643] An "invoice document file" is a file that electronically stores a document used as an invoice, and refers to a digital file in image format, PDF format, or the like.

[0644] "Optical character recognition technology" is a technology that analyzes and extracts character information from images or documents containing handwritten characters, and is also known as OCR (Optical Character Recognition).

[0645] "Generative AI" is a technology that uses machine learning models and artificial intelligence to analyze and generate data, and is particularly used in natural language processing and image analysis.

[0646] The "Qualified Invoice Issuer Database" is a database for registering and managing information on businesses that issue qualified invoices, which allows you to check whether a specific business issues qualified invoices.

[0647] A "cloud platform" is a foundation for using infrastructure such as computing resources and storage provided via the Internet, and is provided as a cloud service.

[0648] "User inquiries" refer to requests made by system users to the system for information or to resolve questions.

[0649] This invention is a system designed to help businesses streamline invoice-related operations. This system functions through the cooperation of three parties: the server, the terminal, and the user. Specifically, it automates everything from uploading invoices, OCR processing, data analysis using AI generation, and determining whether an invoice is eligible, to digitizing and saving the data, and responding to inquiries.

[0650] System configuration

[0651] The system is divided into the following main components:

[0652] server

[0653] The server performs many functions, including:

[0654] 1. Receiving billing data: Receives the billing file sent from the user or terminal.

[0655] 2. OCR processing: Extract character data from the received invoice file using optical character recognition technology (e.g., Google Cloud Vision API or Tesseract OCR).

[0656] 3. Data analysis using generative AI: A generative AI model (e.g., OpenAI GPT-4) is used to parse and extract each item of the invoice from the OCR results.

[0657] 4. Matching with the database of qualified invoice issuers: The extracted issuer information is matched with the database to determine whether the invoice is a qualified invoice.

[0658] 5. Electronic data storage: The adjudicated invoice data is stored electronically on a cloud platform (e.g., AWS S3).

[0659] 6. Response to inquiries using a QA system: Using generative AI, we analyze user inquiries and generate appropriate responses.

[0660] Terminal

[0661] The terminal provides the following features:

[0662] 1. User Interface: Provides an interface for users to upload invoices, check matching results, and make inquiries.

[0663] 2. Data transmission and reception: Sends billing files and inquiry details to the server and receives responses from the server.

[0664] User

[0665] The user does the following:

[0666] 1. Upload invoice: Upload an image or PDF file of your invoice using our dedicated application or web interface.

[0667] 2. Check the matching results: Check the matching results generated by the server and the electronic invoice data.

[0668] 3. Inquiry: Submitting a request to the system for information or to resolve a question.

[0669] Specific examples

[0670] For example, if a business wants to digitize invoices and determine which invoices are eligible, it would follow the steps below:

[0671] 1. The user uses a terminal to upload an image of the invoice to the system. Specifically, typically, the user takes a photo of the invoice using the camera function of a smartphone and uploads the image file.

[0672] 2. The terminal sends the bill file to the server.

[0673] 3. The server receives the file and extracts text data from the invoice using OCR processing, using technologies such as Google Cloud Vision API and Tesseract OCR.

[0674] 4. The server uses a generative AI model (such as OpenAI GPT-4) to analyze and extract each item of the invoice (invoice number, issue date, amount, issuer information, etc.) from the OCR results.

[0675] 5. The server checks the issuer information against the database of qualified invoice issuers and determines that the invoice is a qualified invoice.

[0676] 6. The server electronically stores the adjudicated invoice data on a cloud platform (e.g., AWS S3).

[0677] 7. Users can access the stored data to check the matching results, and if they make inquiries, they can receive a prompt and appropriate response from the generating AI.

[0678] Prompt Sentence Examples

[0679] An example of an input to a generative AI model might be a prompt like this:

[0680] "A user has used their device to upload an image of an invoice. What are the next steps?"

[0681] By using this system, invoice processing operations can be automated, improving work efficiency and accuracy. Furthermore, it becomes possible to respond to user inquiries quickly and accurately.

[0682] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0683] Step 1: Upload your invoice

[0684] The user uploads an image or PDF file of the invoice to the system using the terminal. Specifically, the user clicks the "File Upload" button in the dedicated application or web interface, selects the relevant invoice file, and displays it.

[0685] Input: Invoice image or PDF file.

[0686] Output: The upload request sent to the device.

[0687] Step 2: Submit your invoice file

[0688] The terminal sends the invoice file uploaded by the user to the server using an HTTP POST request, along with the file metadata and user ID.

[0689] Input: User selected invoice file and metadata.

[0690] Output: The file sent to the server along with any accompanying information.

[0691] Step 3: Save the file temporarily

[0692] The server stores the received invoice file in a temporary storage area (for example, an Amazon S3 bucket or local storage). The file name includes the user ID and a timestamp to make it uniquely identifiable.

[0693] Input: Invoice file sent from the terminal.

[0694] Output: Temporarily saved file.

[0695] Step 4: OCR processing

[0696] The server scans the invoice file using OCR technology (Google Cloud Vision API and Tesseract OCR) and extracts text data from the image. After OCR processing, the extracted text data is stored in a temporary database.

[0697] Input: Temporarily saved invoice file.

[0698] Output: The extracted text data.

[0699] Step 5: Temporarily save the text data

[0700] The server stores the text data obtained through the OCR process in a temporary database, and checks the data integrity in the process, specifically logging any incomplete or erroneous data in case it needs to be reprocessed.

[0701] Input: Text data extracted by OCR processing.

[0702] Output: Temporarily saved text data and processing log.

[0703] Step 6: Data analysis and item extraction using generative AI

[0704] The server uses a generative AI model (such as OpenAI GPT-4) to parse and extract each item of the invoice from the OCR results (invoice number, issue date, amount, issuer information, etc.), specifically identifying important entities and organizing them into appropriate fields in a database.

[0705] Input: Temporarily saved text data.

[0706] Output: Structured item data.

[0707] Step 7: Save the analysis data

[0708] The server stores the analyzed and extracted data in this database as structured data in JSON or SQL table format to facilitate subsequent query processing.

[0709] Input: Item data analyzed and extracted by the generative AI.

[0710] Output: Structured data stored in this database.

[0711] Step 8: Match eligible invoice issuers

[0712] The server accesses a database of eligible bill issuers and matches the extracted issuer information by searching the issuer information using a query to determine whether there is a match.

[0713] Input: Structured issuer information.

[0714] Output: Issuer match result.

[0715] Step 9: Update the results

[0716] Based on the result of the matching, the server determines whether the invoice is a qualified invoice or not, and updates the result along with the structured data, adding a determination flag and status information.

[0717] Input: Issuer match result.

[0718] Output: Updated decision result data.

[0719] Step 10: Digitization and storage

[0720] The server generates the final invoice data as electronic data and stores it securely on a cloud platform (such as AWS S3). The data is also encrypted when stored.

[0721] Input: Judgment result data.

[0722] Output: Electronic data stored on a cloud platform.

[0723] Step 11: Sending a save completion notification

[0724] The server generates a notification of completion of saving and sends it to the terminal. The notification includes the URL of the invoice data and status information.

[0725] Input: Saved invoice data.

[0726] Output: Notifications sent to the device.

[0727] Step 12: Submitting an inquiry

[0728] A user uses a terminal to make an inquiry to the system. Specifically, the user enters the inquiry details in the input form and clicks the send button.

[0729] Input: Enquiry details.

[0730] Output: The query request generated on the terminal.

[0731] Step 13: Sending the query request to the server

[0732] The terminal converts the user's inquiry into JSON format and sends it to the server via an HTTP POST request.

[0733] Input: User's inquiry.

[0734] Output: The request sent to the server.

[0735] Step 14: Generate and send a response using the generative AI

[0736] The server uses generative AI (such as OpenAI GPT-4) to analyze the query and generate an appropriate answer, which is then sent to the device and served to the user.

[0737] Input: User's inquiry.

[0738] Output: The response sent to the terminal.

[0739] This concludes the detailed explanation of each processing step of the system. By implementing this system, invoice processing operations can be automated and made more efficient.

[0740] (Application example 1)

[0741] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0742] In conventional logistics centers, product inspection work is done manually, which is inefficient and prone to human error. Furthermore, inspectors need a great deal of prior knowledge and experience to determine whether a product is suitable. Furthermore, responding to inquiries is also manual, which often results in a long response time. There is a need to solve these problems and improve the operational efficiency of logistics centers.

[0743] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0744] In this invention, the server includes a means for receiving an image file of an invoice or label, a means for converting the received file into character data using OCR technology, and a means for analyzing the converted character data using a generation AI to extract each item. This enables the automation of product inspection work at logistics centers and the determination of qualified products.

[0745] "Invoice or label image file" means a scanned image or digital file saved in PDF format of a label or invoice for a product handled at a distribution center or other operation.

[0746] "OCR technology" is an abbreviation for optical character recognition technology, which extracts character information such as letters and numbers from image files and converts them into digital data.

[0747] "Generative AI" is a type of artificial intelligence that uses machine learning algorithms to analyze large amounts of data and generate new information and answers.

[0748] The "eligibility determination database" is a database for collating extracted issuer information or product information to determine its eligibility.

[0749] "Cloud infrastructure" refers to cloud computing infrastructure provided via the Internet, and is a platform for storing, processing, and managing data.

[0750] An "enquiry" is a question or question posed by a user to the system for which the system will generate an answer.

[0751] System Overview

[0752] This invention is a system for automating product inspection work at logistics centers. The system uses smart glasses to scan product labels, analyzes the information in real time, determines whether the product is eligible, and responds to inquiries as needed. The information is then securely stored on a cloud platform.

[0753] System configuration

[0754] Smart Glasses

[0755] The product label is scanned and the image data is sent to the server.

[0756] server

[0757] The received image data is converted into character data using OCR technology.

[0758] The generated text data is analyzed using generative AI to extract each item.

[0759] The extracted product information is compared with the eligibility assessment database, and the assessment results are stored on a cloud platform.

[0760] Generative AI is used to generate responses to user inquiries.

[0761] Cloud infrastructure

[0762] The determined data is stored securely and provided so that the user can access it.

[0763] Specific implementation steps

[0764] 1. Initiate inspection and scan:

[0765] A user scans a product label using smart glasses.

[0766] The smart glasses send the scanned image data to a server.

[0767] 2. Data extraction using OCR processing:

[0768] The server converts the received image data into text data using OCR technology (for example, Google Cloud Vision API).

[0769] The converted character data is temporarily saved.

[0770] 3. Data analysis and eligibility judgment by generative AI:

[0771] The server uses a generative AI model (e.g., OpenAI GPT model) to analyze the text data and extract each item, such as product code, quantity, and expiration date.

[0772] The extracted product information is compared with an eligibility determination database to determine the product's eligibility.

[0773] The determined data is stored on a cloud platform (e.g., AWS S3).

[0774] 4. QA Support:

[0775] When a user makes a query, the server uses generative AI to analyze the query and generate an appropriate answer.

[0776] The server provides the generated answer to the user.

[0777] Specific examples

[0778] Warehouse workers at the distribution center use smart glasses to scan the labels on incoming products. The server analyzes the data in real time and immediately determines whether the product is eligible. If it is eligible, the evidence is stored in AWS S3. If it is not eligible, the system notifies the user of the appropriate measures. Inquiries are also handled promptly using the generative AI model.

[0779] Example prompts to input to a generative AI model:

[0780] "Analyze the information on the product label scanned by the smart glasses, extract data such as the product code, quantity, and expiration date, and then use that information to determine whether the product is eligible. Also, generate an appropriate response to inquiries from users. If the product is eligible, store the data in a cloud platform and notify the user."

[0781] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0782] Step 1:

[0783] A user scans a product label using smart glasses.

[0784] Input: A user puts on the smart glasses and captures an image of a product label.

[0785] Output: The captured image data.

[0786] What it does: When the smart glasses see a product label, they activate their camera and scan the image of the label.

[0787] Step 2:

[0788] The smart glasses send the scanned image data to a server.

[0789] Input: Scanned product label image.

[0790] Output: Image data sent to the server.

[0791] How it works: The smart glasses upload the label image data to the server via a wireless network.

[0792] Step 3:

[0793] The server converts the received image data into text data using OCR technology.

[0794] Input: Received label image data.

[0795] Output: Character data extracted by OCR technology.

[0796] What it does: The server uses an OCR service such as Google Cloud Vision API to extract text information from the label image.

[0797] Step 4:

[0798] The server uses a generative AI model to analyze the converted text data and extract each item.

[0799] Input: Character data obtained by OCR processing.

[0800] Output: Information about each item, such as product code, quantity, and expiration date.

[0801] How it works: The server uses the OpenAI GPT model to analyze the text data and identify and extract the required data items.

[0802] Step 5:

[0803] The server compares the extracted product information with an eligibility determination database to determine eligibility.

[0804] Input: Extracted product information (product code, quantity, expiration date, etc.).

[0805] Output: Product eligibility determination result.

[0806] What happens: The server checks the product information against the eligibility database to determine if the product meets the criteria.

[0807] Step 6:

[0808] The server stores the determined data on a cloud platform.

[0809] Input: Data for the determined eligible product.

[0810] Output: Eligible data stored in a cloud infrastructure.

[0811] What it does: The server stores the data in cloud storage such as AWS S3, making it available for users to access later.

[0812] Step 7:

[0813] The server uses a generation AI to generate responses to user inquiries.

[0814] Input: User's inquiry.

[0815] Output: The generated response to the query.

[0816] Specific operation: Using a generative AI model (e.g., OpenAI GPT model), the query content is analyzed, an optimal answer is generated, and provided to the user.

[0817] In this way, by carrying out specific processing at each step, product inspection work at the logistics center is automated efficiently and accurately.

[0818] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0819] An embodiment of the present invention will be described below as a specific example.

[0820] System Overview

[0821] This invention is a system that allows businesses to improve the efficiency of their invoice-related operations by scanning invoices, determining whether they are qualified, digitizing them, storing them, responding to inquiries, and recognizing user emotions to optimize responses. This system works in cooperation with the server, terminals, and users, and makes use of a variety of technologies.

[0822] System configuration

[0823] server

[0824] Receiving invoice data

[0825] Text extraction using OCR processing

[0826] Data analysis and item extraction using generative AI

[0827] Matching with the database of eligible bill issuers

[0828] Electronic data storage based on the judgment results

[0829] Responding to inquiries through the QA system

[0830] Emotional analysis of users using an emotion engine

[0831] Response optimization based on emotional data

[0832] Learning database updates

[0833] Terminal

[0834] Providing a user interface

[0835] Sending and receiving data to the server

[0836] User

[0837] Uploading an invoice

[0838] Checking the matching results

[0839] Inquiry for necessary information

[0840] Program processing explanation

[0841] Uploading and receiving invoices

[0842] Processing content

[0843] A user uploads an image or PDF file of the invoice to the system using a terminal.

[0844] The terminal stores the uploaded invoice file in a temporary storage area and transmits it to the server.

[0845] The server stores the received bill file in a temporary storage area.

[0846] Invoice scanning and OCR processing

[0847] Processing content

[0848] The server scans the received invoice file using OCR technology.

[0849] The server extracts text data from the invoice using OCR processing and stores it in a temporary database.

[0850] Data analysis and item extraction using generative AI

[0851] Processing content

[0852] The server uses the generative AI model to analyze and extract each item of the invoice (e.g., invoice number, issue date, amount, issuer information, etc.) from the OCR results.

[0853] The server stores the analyzed and extracted data in a database as structured data.

[0854] Verification of eligible invoice issuers

[0855] Processing content

[0856] The server accesses a database of eligible bill issuers and verifies the extracted issuer information.

[0857] Based on the result of the comparison, the server determines whether or not the invoice is a qualified invoice.

[0858] The server updates the result of the determination with the structured data.

[0859] Digitization and storage

[0860] Processing content

[0861] The server then generates the final invoice data as electronic data and stores it on a cloud platform.

[0862] The server notifies the terminal that the saving is complete.

[0863] QA Support and Emotion Engine

[0864] Processing content

[0865] The user makes a query using the terminal.

[0866] The terminal transmits the inquiry to the server.

[0867] The server uses generative AI to analyze the inquiry and generate an appropriate response.

[0868] The server uses an emotion engine as part of the QA system to analyze the user's emotions.

[0869] The server adjusts the content and tone of the response based on the analyzed emotional data to generate the optimal response.

[0870] The server sends the generated answer to the terminal and provides it to the user.

[0871] The server stores the emotional data in a learning database and uses it to improve the response accuracy of the generation AI.

[0872] Specific examples

[0873] For example, when a business digitizes invoices and determines whether they are qualified invoices, the user uploads an image of the invoice to the system from their device. The server then extracts the invoice's text data using OCR processing and uses generation AI to analyze and extract each item on the invoice. The extracted issuer information is compared with a database of qualified invoice issuers, and the invoice is determined to be qualified. Finally, the digitized invoice data is securely stored on a cloud platform, allowing users to access it at any time.

[0874] Additionally, when a user makes an inquiry, the server uses generative AI and an emotion engine to analyze the user's emotions and generate an appropriate response. For example, if the user is feeling anxious or angry, providing a polite response that reflects those emotions can improve user satisfaction. At the same time, this emotional data is stored in a learning database and used to improve the accuracy of future response generation.

[0875] The above is an embodiment of the present invention. This system not only automates bill processing but also improves the user experience.

[0876] The processing flow will be explained below.

[0877] Step 1:

[0878] A user uploads an image or PDF file of the invoice to the system using a terminal.

[0879] Step 2:

[0880] The terminal stores the uploaded invoice file in a temporary storage area and transmits it to the server.

[0881] Step 3:

[0882] The server passes the received invoice file to the OCR processing engine and begins extracting character data.

[0883] Step 4:

[0884] The server retrieves the character data generated by the OCR processing engine and stores it in a temporary text database.

[0885] Step 5:

[0886] The server launches the generation AI, analyzes the saved text data, and extracts each item of the invoice (invoice number, issue date, amount, issuer information, etc.).

[0887] Step 6:

[0888] The server stores each extracted item in a database as structured data.

[0889] Step 7:

[0890] The server accesses a database of eligible bill issuers and verifies the extracted issuer information.

[0891] Step 8:

[0892] Based on the result of the check, the server determines whether the bill is a qualified bill.

[0893] Step 9:

[0894] The server stores the judgment results together with the structured data as electronic data on a cloud platform.

[0895] Step 10:

[0896] The server notifies the terminal that the saving has been completed, and the user is also notified.

[0897] Step 11:

[0898] When a user makes an inquiry using a terminal, the inquiry content is transmitted from the terminal to the server.

[0899] Step 12:

[0900] The server uses generative AI to analyze the inquiry and generate an appropriate response.

[0901] Step 13:

[0902] The server uses an emotion engine as part of the QA system to analyze the user's emotions.

[0903] Step 14:

[0904] The server adjusts the content and tone of the response based on the analyzed emotional data to generate the optimal response.

[0905] Step 15:

[0906] The server sends the generated answer to the terminal and provides it to the user.

[0907] Step 16:

[0908] The server stores the emotional data in a learning database and uses it to improve the response accuracy of the generation AI.

[0909] Step 17:

[0910] The server performs daily backups of all data stored on the cloud platform to ensure data safety.

[0911] Step 18:

[0912] The server performs access control based on user privileges and maintains appropriate security for stored data.

[0913] The above is the specific processing flow in the system of the present invention, which automates invoice processing, significantly reducing the workload of business operators and improving the user experience.

[0914] Example 2

[0915] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0916] Conventional invoice processing systems have the problem of requiring a great deal of manual work and time for scanning, analyzing, judging, digitally storing invoices, and responding to user inquiries. Furthermore, they can only provide a uniform response to user inquiries, making it difficult to improve user satisfaction. To solve these problems, there is a need for a system that can fully automate invoice processing and generate appropriate responses based on the user's emotions.

[0917] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0918] In this invention, the server includes means for receiving an image or electronic file of an invoice, means for converting the received file into character data using optical character recognition technology, means for analyzing the converted character data using a generative AI model and extracting each item of the invoice, means for comparing the extracted invoice issuer information with a database of qualified invoice issuers, means for determining whether the invoice is a qualified invoice based on the comparison results, means for storing the determined invoice as electronic data on a cloud platform, means for generating responses to inquiries from users using the generative AI model, means for using an emotion analysis engine to analyze user emotions and adjust the content and tone of the response, and means for saving the emotion analysis data in a learning database and improving the response accuracy of the generative AI model. This enables full automation of invoice processing operations and the generation of appropriate responses according to user emotions.

[0919] 1. "Invoice" means a document used to request payment for goods or services.

[0920] 2. "Image" means any form in which visual data can be stored and displayed electronically.

[0921] 3. "Electronic files" refers to documents and data stored and managed in digital form.

[0922] 4. "Receiving" refers to obtaining data or information from an external source.

[0923] 5. "Optical character recognition technology" is a technology that analyzes letters and numbers in an image and converts them into text data.

[0924] 6. “Generative AI model” means a model that uses artificial intelligence to generate and analyze data.

[0925] 7. "Analysis" refers to examining data in detail to clarify its content and structure.

[0926] 8. "Item" means an individual element that refers to a specific piece of information or data.

[0927] 9. "Qualified Bill Issuer Database" is a database for verifying the eligibility of bill issuers.

[0928] 10. "Matching" refers to comparing different data to determine whether they match or fit.

[0929] 11. "Judgment" means determining suitability based on criteria.

[0930] 12. "Electronic data" means data stored and managed in digital form.

[0931] 13. "Cloud infrastructure" refers to infrastructure with data storage and processing capabilities provided via the Internet.

[0932] 14. "Inquiry" means a question or request for information or an answer.

[0933] 15. "Response" means an answer or reaction provided to an inquiry or request.

[0934] 16. "Sentiment analysis engine" refers to software technology for analyzing emotions from data such as text and voice.

[0935] 17. "Training Database" means a database that stores and manages data used to train machine learning models.

[0936] This invention is a system for streamlining invoice-related tasks, and it works in cooperation with the server, terminal, and user. Specifically, it is equipped with technology for scanning invoices, determining whether they are qualified invoices, digitizing them, storing them, responding to inquiries, and recognizing user emotions to optimize responses.

[0937] System configuration

[0938] Hardware and Software Configuration

[0939] Server: Use a high-performance cloud server (e.g., AWS EC2).

[0940] Device: The computer or smartphone used by the user.

[0941] OCR technology: Uses Google Cloud Vision API.

[0942] Generative AI model: OpenAI GPT-4 is used.

[0943] Cloud infrastructure: AWS S3 is used for data storage.

[0944] Sentiment analysis engine: Uses Microsoft Azure Emotion API.

[0945] Specific explanation of program processing

[0946] Uploading and receiving invoices

[0947] Users upload invoice images or PDF files to the system using a dedicated web interface on their device or a smartphone application. The device temporarily stores the uploaded file in local storage and then sends it to the server using HTTPS. The server then stores the file in a temporary storage area (AWS S3 bucket).

[0948] Invoice scanning and OCR processing

[0949] The server scans the received invoice file using Google Cloud Vision API and extracts the text data, which is then stored in a temporary database (MongoDB).

[0950] Data analysis and item extraction using generative AI

[0951] The server uses OpenAI GPT-4 to analyze and extract each item of the invoice (e.g., invoice number, issue date, amount, issuer information, etc.) from the OCR processing results. The analysis is performed by inputting the following prompt sentence into the generation AI:

[0952] Extract the invoice items from this text.

[0953] The parsed and extracted data is stored in a database (MySQL) as structured data (e.g., database records).

[0954] Verification of eligible invoice issuers

[0955] The server accesses the database of qualified invoice issuers (a database of the National Tax Agency) and compares the issuer information extracted by the generation AI. Based on the comparison results, the server determines that the invoice is a qualified invoice and updates the internal database.

[0956] Digitization and storage

[0957] The server generates the final invoice data as electronic data (e.g. PDF format) and saves it on the cloud infrastructure (AWS S3). It notifies the terminal that saving is complete.

[0958] QA Support and Emotion Engine

[0959] The user enters a question into a dedicated inquiry form and presses the send button. The device sends the information to the server, which uses OpenAI GPT-4 to analyze the inquiry and generate an appropriate answer. The server also analyzes the user's emotions using the Microsoft Azure Emotion API and adjusts the content and tone of the response. For example, the following prompt sentences can be used:

[0960] I have a question regarding invoice number 12345. The invoice was determined to be ineligible. I would like to know the reason for this and how I can resubmit it as an eligible invoice.

[0961] The final answer generated is provided to the user, and the sentiment analysis data is stored in a learning database to help improve the accuracy of future response generation.

[0962] Specific examples

[0963] For example, when a business digitizes invoices and determines whether they are qualified invoices, the user uploads an image of the invoice to the system from their device. The server uses OCR processing to extract the text data from the invoice, and uses generation AI to analyze and extract each item on the invoice. The extracted issuer information is compared with a database of qualified invoice issuers, and the invoice is determined to be qualified. The digitized invoice data is securely stored on a cloud platform, allowing users to access it at any time.

[0964] Additionally, when a user makes an inquiry, the server uses generative AI and an emotion engine to analyze the user's emotions and generate an appropriate response. For example, if a user is feeling anxious or angry, providing a polite response that reflects those emotions can improve user satisfaction. At the same time, this emotional data is stored in a learning database and used to improve the accuracy of future response generation. This system significantly improves the efficiency of invoice processing and the user experience.

[0965] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0966] Step 1: Upload your invoice

[0967] Users upload invoice images or PDF files to the system using a dedicated web interface or smartphone app. The input is the invoice image file or PDF file, and the output is saving it to the terminal's temporary storage area. The terminal temporarily saves the uploaded file in local storage and sends it to the server using the HTTPS protocol.

[0968] Step 2: Receiving and temporarily saving the file

[0969] The server receives the invoice file sent from the terminal and saves it in a temporary storage area within the cloud infrastructure (e.g. AWS S3). The input is the invoice file sent from the terminal, and the output is the file saved in the temporary storage area within the cloud infrastructure. The file is checked for integrity and scanned for viruses, and if there are no problems, it proceeds to the next processing step.

[0970] Step 3: OCR the invoice

[0971] The server scans the invoice file stored in the temporary storage area of ​​the cloud infrastructure using the Google Cloud Vision API. The input is the invoice file stored in the temporary storage area, and the output is text data. Specifically, the letters and numbers in the image are extracted as text data, and the data is stored in a temporary database (e.g., MongoDB).

[0972] Step 4: Data analysis with generative AI

[0973] The server uses OpenAI GPT-4 to analyze and extract each item of the invoice based on the extracted character data. The input is the extracted character data, and the output is structured data including each item of the invoice (e.g., invoice number, issue date, amount, issuer information, etc.). The following prompt sentence is input to the generation AI:

[0974] Extract the invoice items from this text.

[0975] The generated structured data is stored in a database (e.g., MySQL).

[0976] Step 5: Verify eligible billing entities

[0977] The server compares the issuer information stored in the database with the database of qualified invoice issuers (a database of the National Tax Agency). The input is the issuer information, and the output is the eligibility determination result. Specifically, the server compares the issuer information with the database and determines whether the invoice is appropriate based on the results.

[0978] Step 6: Save the relevance data

[0979] The server updates the database with structured data that reflects the judgment result. The input is the eligibility judgment result, and the output is the updated structured data.

[0980] Step 7: Electronic data generation and final storage

[0981] The server generates the final invoice data that has been determined to be eligible as electronic data and stores it on a cloud platform (e.g., AWS S3). The input is the determined invoice data, and the output is the final electronic data. A notification that the data has been saved is then sent to the terminal.

[0982] Step 8: Receiving inquiries from users

[0983] The user enters a question into a dedicated inquiry form and presses the send button. The input is the inquiry content, and the output is the transmission of the inquiry content to the server. The terminal then sends the inquiry content to the server.

[0984] Step 9: Parsing the query and generating a response

[0985] The server analyzes the query using OpenAI GPT-4 and generates an appropriate answer. The input is the query, and the output is the generated answer. It also analyzes the user's emotions using Microsoft Azure Emotion API and adjusts the content and tone of the response. For example, it uses the following prompt:

[0986] I have a question regarding invoice number 12345. The invoice was determined to be ineligible. I would like to know the reason for this and how I can resubmit it as an eligible invoice.

[0987] Step 10: Storing emotion data and serving responses

[0988] The server stores the generated answer together with the emotion analysis data in a learning database and transmits the answer to the terminal. The input is the generated answer and the emotion analysis data, and the output is saving the answer in the learning database and providing the answer to the user.

[0989] The above is the specific flow of operations in the system's processing steps. This system will greatly improve the efficiency of invoice processing and the user experience.

[0990] (Application example 2)

[0991] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0992] Improving customer service efficiency and customer satisfaction are currently key challenges for many brick-and-mortar stores. In particular, quick and accurate responses are required when handling inquiries and troubleshooting, which directly impacts the customer experience. However, handling a large number of inquiries requires human intervention, making it difficult to secure and manage a workforce. Furthermore, while properly understanding customer emotions and responding accordingly could contribute to further improving customer satisfaction, automation remains lacking in this area.

[0993] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for receiving an image or electronic document of an invoice; means for converting the received document into character data using optical character recognition technology; means for analyzing the converted character data using generative artificial intelligence and extracting each item of the invoice; means for comparing the extracted invoice issuer information with a database of qualified invoice issuers; means for determining whether the invoice is a qualified invoice based on the comparison result; means for storing the determined invoice as electronic data on a remote platform; means for generating a response to a user inquiry using generative artificial intelligence; and means for adjusting the response based on the emotion data using an emotion recognition engine that analyzes the user's emotions. This makes it possible for brick-and-mortar stores to automatically provide optimal responses based on the analysis of receipt and invoice information and the customer's emotions.

[0994] "Invoice image or electronic document" refers to an image file or a digital document such as a PDF that visually represents the contents of the invoice.

[0995] "Optical character recognition technology" is a technology for analyzing character information contained in an image and converting it into text data.

[0996] "Generative artificial intelligence" is an artificial intelligence technology that generates new data and responses based on large amounts of data, and is used for natural language generation and various data analyses.

[0997] The "Qualified Invoice Issuer Database" is a database that compiles information on businesses that have the authority to issue qualified invoices.

[0998] "Remote infrastructure" refers to cloud environments and remote data storage accessible via the Internet.

[0999] An "emotion recognition engine" is an engine that analyzes and identifies a user's emotional state from data such as text and voice.

[1000] "User emotion" refers to the user's psychological state and emotional expression during an inquiry or conversation.

[1001] "Response" or "response generation" is the process of automatically creating an appropriate reply to a user's inquiry using generative artificial intelligence.

[1002] System Overview

[1003] The system of the present invention includes a series of processes, from reading invoices to determining whether they are eligible, converting them into electronic data, storing them, responding to inquiries, and recognizing the user's emotions to generate the most appropriate response. This system works in cooperation with the server, terminals, and users, and makes full use of a variety of technologies, including OCR technology, generative artificial intelligence, and an emotion recognition engine.

[1004] Server Features

[1005] The server has the following main functions:

[1006] 1. Receiving invoices: Receiving invoice images and electronic documents uploaded by users.

[1007] 2. Optical character recognition processing: The received document is converted into character data using optical character recognition technology (such as pytesseract).

[1008] 3. Data analysis and item extraction: Using generative artificial intelligence (such as Hugging Face's transformers library), each item of the invoice (e.g., invoice number, issue date, amount, issuer information, etc.) is analyzed and extracted from the converted text data.

[1009] 4. Database matching: The extracted invoice issuer information is matched with the database of eligible invoice issuers to determine whether the invoice is eligible.

[1010] 5. Digitalization and storage: The judged invoices are stored as electronic data on a cloud platform.

[1011] 6. Inquiry handling and response generation: Generative artificial intelligence is used to generate appropriate responses to user inquiries.

[1012] 7. Sentiment Analysis: An emotion recognition engine is used to analyze the user's emotional data and tailor the response.

[1013] Device Features

[1014] The terminal provides the following functions to the user:

[1015] 1. Provide an interface: Through the user interface, you can upload invoices, send and receive inquiries, etc.

[1016] 2. Sending and receiving data: Two-way communication with the server to send or receive data.

[1017] User operations

[1018] The user does the following:

[1019] 1. Uploading invoices: Use the terminal to upload an image or electronic document of the invoice to the system.

[1020] 2. Sending a query: Use the terminal to query the system for the required information.

[1021] 3. Check the matching result: Check the matching result sent from the server.

[1022] Hardware and software used

[1023] The hardware used is a customer-facing robot from a brick-and-mortar store (e.g., a typical customer-facing robot), while the software uses OCR technology (pytesseract), generative artificial intelligence (Hugging Face's transformers library and GPT-2 model), and a sentiment analysis module (sentiment_analysis).

[1024] Specific examples

[1025] For example, if a customer asks "Can I get a refund for this item?" in a store, the robot will analyze the customer's facial expressions and tone of voice to determine their emotions, and if the emotion score is low, it will generate a more polite and flexible response. At the same time, if the customer shows the robot a receipt, it will use OCR technology to quickly read the information on the receipt and immediately determine whether a refund is possible and respond accordingly.

[1026] Prompt Sentence Examples

[1027] User Question: "Can I get a refund for this item?"

[1028] Depending on the sentiment analysis results, the generated response may be:

[1029] "Of course, I'm happy to refund this item." (High sentiment score)

[1030] "Sorry, but we need to go through some checks before we can process your refund. Could you please wait a moment?" (if your sentiment score is low)

[1031] This system is expected to improve the efficiency of customer service in physical stores and increase customer satisfaction.

[1032] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1033] Step 1:

[1034] The server receives the invoice image or electronic document uploaded by the user. The input format can be an image file, a PDF file, or other format. The received file is stored in a temporary storage area.

[1035] Step 2:

[1036] The server converts the received invoice into text data using optical character recognition (OCR). It scans the image or PDF file of the invoice received as input and generates text data as output. Specifically, it uses pytesseract to analyze the characters in the image or PDF and convert them into text format.

[1037] Step 3:

[1038] The server uses generative artificial intelligence to analyze and extract each item of the invoice (e.g., invoice number, issue date, amount, issuer information, etc.) from the converted character data. It uses text data generated by OCR as input and generates structured data containing each item of the invoice as output. Specifically, it uses a generative artificial intelligence model using Hugging Face's transformers library.

[1039] Step 4:

[1040] The server compares the extracted invoice issuer information with the database of qualified invoice issuers. It uses the issuer information and database information as input and obtains the matching result as output. This completes the process of determining whether the invoice is qualified.

[1041] Step 5:

[1042] Based on the judgment results, the server stores qualified invoices as electronic data on a cloud platform. Using the matching results and invoice data as input, it generates a message indicating that the data has been saved as output. The electronic data is securely stored in cloud storage and can be accessed by users.

[1043] Step 6:

[1044] When a user makes a query using a terminal, the server receives the query content. At this time, the query content is sent as input and stored on the server side for processing.

[1045] Step 7:

[1046] The server uses generative artificial intelligence to generate appropriate responses to user queries. It uses the user's query as input and obtains the generated response text as output. Specifically, it uses Hugging Face's transformers library to generate the optimal response corresponding to the query.

[1047] Step 8:

[1048] The server uses an emotion recognition engine to analyze the user's emotion data. It uses the query content, the user's tone of voice, and text data as input, and generates an emotion score as output. It uses a sentiment analysis module (such as Sentiment Analysis).

[1049] Step 9:

[1050] The server adjusts the content and tone of the response based on the analyzed emotion data. It uses the emotion score and the initial response text as input and generates an optimized response text as output, which provides an appropriate response that corresponds to the user's emotion.

[1051] Step 10:

[1052] The server sends the generated response to the terminal and provides it to the user, using the optimized response text as input and displaying it on the terminal as output, after which the user can check the generated response.

[1053] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1054] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1055] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1056] [Third embodiment]

[1057] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1058] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1059] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1060] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1061] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1062] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1063] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1064] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1065] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1066] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1067] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1068] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1069] An embodiment of the present invention will be described below as a specific example.

[1070] System Overview

[1071] This invention is a system that automates everything from reading invoices to determining whether they are qualified invoices, digitizing them, storing them, and responding to inquiries, in order to streamline invoice-related operations for businesses. This system works in cooperation with three parties: a server, a terminal, and a user.

[1072] System configuration

[1073] server

[1074] Receiving invoice data

[1075] Text extraction using OCR processing

[1076] Data analysis and item extraction using generative AI

[1077] Matching with the database of eligible bill issuers

[1078] Electronic data storage based on the judgment results

[1079] Responding to inquiries through the QA system

[1080] Terminal

[1081] Providing a user interface

[1082] Sending and receiving data to the server

[1083] User

[1084] Uploading an invoice

[1085] Checking the matching results

[1086] Inquiry for necessary information

[1087] Program processing explanation

[1088] Uploading and receiving invoices

[1089] Processing content

[1090] A user uploads an image or PDF file of the invoice to the system using a terminal.

[1091] The terminal sends the uploaded bill file to the server.

[1092] The server stores the received bill file in a temporary storage area.

[1093] Invoice scanning and OCR processing

[1094] Processing content

[1095] The server scans the received invoice file using OCR technology.

[1096] The server extracts text data from the invoice using OCR processing and stores it in a temporary database.

[1097] Data analysis and item extraction using generative AI

[1098] Processing content

[1099] The server uses the generative AI model to analyze and extract each item of the invoice (e.g., invoice number, issue date, amount, issuer information, etc.) from the OCR results.

[1100] The server stores the analyzed and extracted data in a database as structured data.

[1101] Verification of eligible invoice issuers

[1102] Processing content

[1103] The server accesses a database of eligible bill issuers and verifies the extracted issuer information.

[1104] Based on the result of the comparison, the server determines whether or not the invoice is a qualified invoice.

[1105] The server updates the result of the determination with the structured data.

[1106] Digitization and storage

[1107] Processing content

[1108] The server then generates the final invoice data as electronic data and stores it on a cloud platform.

[1109] The server notifies the terminal that the saving is complete.

[1110] QA Support

[1111] Processing content

[1112] The user makes a query using the terminal.

[1113] The terminal transmits the inquiry to the server.

[1114] The server uses generative AI to analyze the inquiry and generate an appropriate response.

[1115] The server sends the generated answer to the terminal and provides it to the user.

[1116] Specific examples

[1117] For example, when a business digitizes invoices and determines whether they are qualified invoices, the user uploads an image of the invoice to the system from their device. The server then extracts the invoice's text data using OCR processing, and uses generation AI to analyze and extract each item on the invoice. The extracted issuer information is compared with a database of qualified invoice issuers, and the invoice is determined to be qualified. Finally, the digitized invoice data is securely stored on a cloud platform, allowing users to access it at any time. Furthermore, if the user makes an inquiry, the generation AI can respond quickly.

[1118] The above is an embodiment of the present invention. This system automates invoice processing and significantly improves operational efficiency.

[1119] The processing flow will be explained below.

[1120] Step 1:

[1121] A user uploads an image or PDF file of the invoice to the system using a terminal.

[1122] Step 2:

[1123] The terminal stores the uploaded invoice file in a temporary storage area and transmits it to the server.

[1124] Step 3:

[1125] The server passes the received invoice file to the OCR processing engine and begins extracting character data.

[1126] Step 4:

[1127] The server retrieves the character data generated by the OCR processing engine and stores it in a temporary text database.

[1128] Step 5:

[1129] The server launches the generation AI, analyzes the saved text data, and extracts each item of the invoice (invoice number, issue date, amount, issuer information, etc.).

[1130] Step 6:

[1131] The server stores each extracted item in a database as structured data.

[1132] Step 7:

[1133] The server accesses a database of eligible bill issuers and verifies the extracted issuer information.

[1134] Step 8:

[1135] Based on the result of the check, the server determines whether the bill is a qualified bill.

[1136] Step 9:

[1137] The server stores the judgment results together with the structured data as electronic data on a cloud platform.

[1138] Step 10:

[1139] The server notifies the terminal that the saving has been completed, and the user is also notified.

[1140] Step 11:

[1141] When a user makes an inquiry using a terminal, the inquiry content is transmitted from the terminal to the server.

[1142] Step 12:

[1143] The server uses generative AI to analyze the inquiry and generate an appropriate response.

[1144] Step 13:

[1145] The server sends the generated answer to the terminal and provides it to the user.

[1146] Step 14:

[1147] The server performs daily backups of all data stored on the cloud platform to ensure data safety.

[1148] Step 15:

[1149] The server performs access control based on user privileges and maintains appropriate security for stored data.

[1150] The above is the specific processing flow in the system of the present invention, which automates the invoice processing work and significantly reduces the workload of business operators.

[1151] Example 1

[1152] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1153] Traditional invoice processing operations require a lot of manual input and verification, which takes time and effort. There is also a high risk of human error, making it necessary to streamline operations. Furthermore, delays in determining and responding to qualified invoices can expose businesses to legal risks and a decline in their credibility. Furthermore, the lack of a system for quickly responding to user inquiries makes it difficult to provide reliable services.

[1154] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1155] In this invention, the server includes means for receiving an invoice document file, means for converting the received file into character data using optical character recognition technology, means for analyzing the converted character data using a generation AI and extracting each item of the invoice, means for comparing the extracted invoice issuer information with a qualified invoice issuer database, means for determining whether the invoice is a qualified invoice based on the comparison results, means for storing the determined invoice as electronic data on a cloud platform, and means for generating responses to user inquiries using a generation AI. This reduces manual work, improves business efficiency and accuracy, and enables early determination of qualified invoices and secure electronic storage. Furthermore, user inquiries can be responded to quickly and accurately.

[1156] An "invoice document file" is a file that electronically stores a document used as an invoice, and refers to a digital file in image format, PDF format, or the like.

[1157] "Optical character recognition technology" is a technology that analyzes and extracts character information from images or documents containing handwritten characters, and is also known as OCR (Optical Character Recognition).

[1158] "Generative AI" is a technology that uses machine learning models and artificial intelligence to analyze and generate data, and is particularly used in natural language processing and image analysis.

[1159] The "Qualified Invoice Issuer Database" is a database for registering and managing information on businesses that issue qualified invoices, which allows you to check whether a specific business issues qualified invoices.

[1160] A "cloud platform" is a foundation for using infrastructure such as computing resources and storage provided via the Internet, and is provided as a cloud service.

[1161] "User inquiries" refer to requests made by system users to the system for information or to resolve questions.

[1162] This invention is a system designed to help businesses streamline invoice-related operations. This system functions through the cooperation of three parties: the server, the terminal, and the user. Specifically, it automates everything from uploading invoices, OCR processing, data analysis using AI generation, and determining whether an invoice is eligible, to digitizing and saving the data, and responding to inquiries.

[1163] System configuration

[1164] The system is divided into the following main components:

[1165] server

[1166] The server performs many functions, including:

[1167] 1. Receiving billing data: Receives the billing file sent from the user or terminal.

[1168] 2. OCR processing: Extract character data from the received invoice file using optical character recognition technology (e.g., Google Cloud Vision API or Tesseract OCR).

[1169] 3. Data analysis using generative AI: A generative AI model (e.g., OpenAI GPT-4) is used to parse and extract each item of the invoice from the OCR results.

[1170] 4. Matching with the database of qualified invoice issuers: The extracted issuer information is matched with the database to determine whether the invoice is a qualified invoice.

[1171] 5. Electronic data storage: The adjudicated invoice data is stored electronically on a cloud platform (e.g., AWS S3).

[1172] 6. Response to inquiries using a QA system: Using generative AI, we analyze user inquiries and generate appropriate responses.

[1173] Terminal

[1174] The terminal provides the following features:

[1175] 1. User Interface: Provides an interface for users to upload invoices, check matching results, and make inquiries.

[1176] 2. Data transmission and reception: Sends billing files and inquiry details to the server and receives responses from the server.

[1177] User

[1178] The user does the following:

[1179] 1. Upload invoice: Upload an image or PDF file of your invoice using our dedicated application or web interface.

[1180] 2. Check the matching results: Check the matching results generated by the server and the electronic invoice data.

[1181] 3. Inquiry: Submitting a request to the system for information or to resolve a question.

[1182] Specific examples

[1183] For example, if a business wants to digitize invoices and determine which invoices are eligible, it would follow the steps below:

[1184] 1. The user uses a terminal to upload an image of the invoice to the system. Specifically, typically, the user takes a photo of the invoice using the camera function of a smartphone and uploads the image file.

[1185] 2. The terminal sends the bill file to the server.

[1186] 3. The server receives the file and extracts text data from the invoice using OCR processing, using technologies such as Google Cloud Vision API and Tesseract OCR.

[1187] 4. The server uses a generative AI model (such as OpenAI GPT-4) to analyze and extract each item of the invoice (invoice number, issue date, amount, issuer information, etc.) from the OCR results.

[1188] 5. The server checks the issuer information against the database of qualified invoice issuers and determines that the invoice is a qualified invoice.

[1189] 6. The server electronically stores the adjudicated invoice data on a cloud platform (e.g., AWS S3).

[1190] 7. Users can access the stored data to check the matching results, and if they make inquiries, they can receive a prompt and appropriate response from the generating AI.

[1191] Prompt Sentence Examples

[1192] An example of an input to a generative AI model might be a prompt like this:

[1193] "A user has used their device to upload an image of an invoice. What are the next steps?"

[1194] By using this system, invoice processing operations can be automated, improving work efficiency and accuracy. Furthermore, it becomes possible to respond to user inquiries quickly and accurately.

[1195] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1196] Step 1: Upload your invoice

[1197] The user uploads an image or PDF file of the invoice to the system using the terminal. Specifically, the user clicks the "File Upload" button in the dedicated application or web interface, selects the relevant invoice file, and displays it.

[1198] Input: Invoice image or PDF file.

[1199] Output: The upload request sent to the device.

[1200] Step 2: Submit your invoice file

[1201] The terminal sends the invoice file uploaded by the user to the server using an HTTP POST request, along with the file metadata and user ID.

[1202] Input: User selected invoice file and metadata.

[1203] Output: The file sent to the server along with any accompanying information.

[1204] Step 3: Save the file temporarily

[1205] The server stores the received invoice file in a temporary storage area (for example, an Amazon S3 bucket or local storage). The file name includes the user ID and a timestamp to make it uniquely identifiable.

[1206] Input: Invoice file sent from the terminal.

[1207] Output: Temporarily saved file.

[1208] Step 4: OCR processing

[1209] The server scans the invoice file using OCR technology (Google Cloud Vision API and Tesseract OCR) and extracts text data from the image. After OCR processing, the extracted text data is stored in a temporary database.

[1210] Input: Temporarily saved invoice file.

[1211] Output: The extracted text data.

[1212] Step 5: Temporarily save the text data

[1213] The server stores the text data obtained through the OCR process in a temporary database, and checks the data integrity in the process, specifically logging any incomplete or erroneous data in case it needs to be reprocessed.

[1214] Input: Text data extracted by OCR processing.

[1215] Output: Temporarily saved text data and processing log.

[1216] Step 6: Data analysis and item extraction using generative AI

[1217] The server uses a generative AI model (such as OpenAI GPT-4) to parse and extract each item of the invoice from the OCR results (invoice number, issue date, amount, issuer information, etc.), specifically identifying important entities and organizing them into appropriate fields in a database.

[1218] Input: Temporarily saved text data.

[1219] Output: Structured item data.

[1220] Step 7: Save the analysis data

[1221] The server stores the analyzed and extracted data in this database as structured data in JSON or SQL table format to facilitate subsequent query processing.

[1222] Input: Item data analyzed and extracted by the generative AI.

[1223] Output: Structured data stored in this database.

[1224] Step 8: Match eligible invoice issuers

[1225] The server accesses a database of eligible bill issuers and matches the extracted issuer information by searching the issuer information using a query to determine whether there is a match.

[1226] Input: Structured issuer information.

[1227] Output: Issuer match result.

[1228] Step 9: Update the results

[1229] Based on the result of the matching, the server determines whether the invoice is a qualified invoice or not, and updates the result along with the structured data, adding a determination flag and status information.

[1230] Input: Issuer match result.

[1231] Output: Updated decision result data.

[1232] Step 10: Digitization and storage

[1233] The server generates the final invoice data as electronic data and stores it securely on a cloud platform (such as AWS S3). The data is also encrypted when stored.

[1234] Input: Judgment result data.

[1235] Output: Electronic data stored on a cloud platform.

[1236] Step 11: Sending a save completion notification

[1237] The server generates a notification of completion of saving and sends it to the terminal. The notification includes the URL of the invoice data and status information.

[1238] Input: Saved invoice data.

[1239] Output: Notifications sent to the device.

[1240] Step 12: Submitting an inquiry

[1241] A user uses a terminal to make an inquiry to the system. Specifically, the user enters the inquiry details in the input form and clicks the send button.

[1242] Input: Enquiry details.

[1243] Output: The query request generated on the terminal.

[1244] Step 13: Sending the query request to the server

[1245] The terminal converts the user's inquiry into JSON format and sends it to the server via an HTTP POST request.

[1246] Input: User's inquiry.

[1247] Output: The request sent to the server.

[1248] Step 14: Generate and send a response using the generative AI

[1249] The server uses generative AI (such as OpenAI GPT-4) to analyze the query and generate an appropriate answer, which is then sent to the device and served to the user.

[1250] Input: User's inquiry.

[1251] Output: The response sent to the terminal.

[1252] This concludes the detailed explanation of each processing step of the system. By implementing this system, invoice processing operations can be automated and made more efficient.

[1253] (Application example 1)

[1254] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1255] In conventional logistics centers, product inspection work is done manually, which is inefficient and prone to human error. Furthermore, inspectors need a great deal of prior knowledge and experience to determine whether a product is suitable. Furthermore, responding to inquiries is also manual, which often results in a long response time. There is a need to solve these problems and improve the operational efficiency of logistics centers.

[1256] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1257] In this invention, the server includes a means for receiving an image file of an invoice or label, a means for converting the received file into character data using OCR technology, and a means for analyzing the converted character data using a generation AI to extract each item. This enables the automation of product inspection work at logistics centers and the determination of qualified products.

[1258] "Invoice or label image file" means a scanned image or digital file saved in PDF format of a label or invoice for a product handled at a distribution center or other operation.

[1259] "OCR technology" is an abbreviation for optical character recognition technology, which extracts character information such as letters and numbers from image files and converts them into digital data.

[1260] "Generative AI" is a type of artificial intelligence that uses machine learning algorithms to analyze large amounts of data and generate new information and answers.

[1261] The "eligibility determination database" is a database for collating extracted issuer information or product information to determine its eligibility.

[1262] "Cloud infrastructure" refers to cloud computing infrastructure provided via the Internet, and is a platform for storing, processing, and managing data.

[1263] An "enquiry" is a question or question posed by a user to the system for which the system will generate an answer.

[1264] System Overview

[1265] This invention is a system for automating product inspection work at logistics centers. The system uses smart glasses to scan product labels, analyzes the information in real time, determines whether the product is eligible, and responds to inquiries as needed. The information is then securely stored on a cloud platform.

[1266] System configuration

[1267] Smart Glasses

[1268] The product label is scanned and the image data is sent to the server.

[1269] server

[1270] The received image data is converted into character data using OCR technology.

[1271] The generated text data is analyzed using generative AI to extract each item.

[1272] The extracted product information is compared with the eligibility assessment database, and the assessment results are stored on a cloud platform.

[1273] Generative AI is used to generate responses to user inquiries.

[1274] Cloud infrastructure

[1275] The determined data is stored securely and provided so that the user can access it.

[1276] Specific implementation steps

[1277] 1. Initiate inspection and scan:

[1278] A user scans a product label using smart glasses.

[1279] The smart glasses send the scanned image data to a server.

[1280] 2. Data extraction using OCR processing:

[1281] The server converts the received image data into text data using OCR technology (for example, Google Cloud Vision API).

[1282] The converted character data is temporarily saved.

[1283] 3. Data analysis and eligibility judgment by generative AI:

[1284] The server uses a generative AI model (e.g., OpenAI GPT model) to analyze the text data and extract each item, such as product code, quantity, and expiration date.

[1285] The extracted product information is compared with an eligibility determination database to determine the product's eligibility.

[1286] The determined data is stored on a cloud platform (e.g., AWS S3).

[1287] 4. QA Support:

[1288] When a user makes a query, the server uses generative AI to analyze the query and generate an appropriate answer.

[1289] The server provides the generated answer to the user.

[1290] Specific examples

[1291] Warehouse workers at the distribution center use smart glasses to scan the labels on incoming products. The server analyzes the data in real time and immediately determines whether the product is eligible. If it is eligible, the evidence is stored in AWS S3. If it is not eligible, the system notifies the user of the appropriate measures. Inquiries are also handled promptly using the generative AI model.

[1292] Example prompts to input to a generative AI model:

[1293] "Analyze the information on the product label scanned by the smart glasses, extract data such as the product code, quantity, and expiration date, and then use that information to determine whether the product is eligible. Also, generate an appropriate response to inquiries from users. If the product is eligible, store the data in a cloud platform and notify the user."

[1294] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1295] Step 1:

[1296] A user scans a product label using smart glasses.

[1297] Input: A user puts on the smart glasses and captures an image of a product label.

[1298] Output: The captured image data.

[1299] What it does: When the smart glasses see a product label, they activate their camera and scan the image of the label.

[1300] Step 2:

[1301] The smart glasses send the scanned image data to a server.

[1302] Input: Scanned product label image.

[1303] Output: Image data sent to the server.

[1304] How it works: The smart glasses upload the label image data to the server via a wireless network.

[1305] Step 3:

[1306] The server converts the received image data into text data using OCR technology.

[1307] Input: Received label image data.

[1308] Output: Character data extracted by OCR technology.

[1309] What it does: The server uses an OCR service such as Google Cloud Vision API to extract text information from the label image.

[1310] Step 4:

[1311] The server uses a generative AI model to analyze the converted text data and extract each item.

[1312] Input: Character data obtained by OCR processing.

[1313] Output: Information about each item, such as product code, quantity, and expiration date.

[1314] How it works: The server uses the OpenAI GPT model to analyze the text data and identify and extract the required data items.

[1315] Step 5:

[1316] The server compares the extracted product information with an eligibility determination database to determine eligibility.

[1317] Input: Extracted product information (product code, quantity, expiration date, etc.).

[1318] Output: Product eligibility determination result.

[1319] What happens: The server checks the product information against the eligibility database to determine if the product meets the criteria.

[1320] Step 6:

[1321] The server stores the determined data on a cloud platform.

[1322] Input: Data for the determined eligible product.

[1323] Output: Eligible data stored in a cloud infrastructure.

[1324] What it does: The server stores the data in cloud storage such as AWS S3, making it available for users to access later.

[1325] Step 7:

[1326] The server uses a generation AI to generate responses to user inquiries.

[1327] Input: User's inquiry.

[1328] Output: The generated response to the query.

[1329] Specific operation: Using a generative AI model (e.g., OpenAI GPT model), the query content is analyzed, an optimal answer is generated, and provided to the user.

[1330] In this way, by carrying out specific processing at each step, product inspection work at the logistics center is automated efficiently and accurately.

[1331] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1332] An embodiment of the present invention will be described below as a specific example.

[1333] System Overview

[1334] This invention is a system that allows businesses to improve the efficiency of their invoice-related operations by scanning invoices, determining whether they are qualified, digitizing them, storing them, responding to inquiries, and recognizing user emotions to optimize responses. This system works in cooperation with the server, terminals, and users, and makes use of a variety of technologies.

[1335] System configuration

[1336] server

[1337] Receiving invoice data

[1338] Text extraction using OCR processing

[1339] Data analysis and item extraction using generative AI

[1340] Matching with the database of eligible bill issuers

[1341] Electronic data storage based on the judgment results

[1342] Responding to inquiries through the QA system

[1343] Emotional analysis of users using an emotion engine

[1344] Response optimization based on emotional data

[1345] Learning database updates

[1346] Terminal

[1347] Providing a user interface

[1348] Sending and receiving data to the server

[1349] User

[1350] Uploading an invoice

[1351] Checking the matching results

[1352] Inquiry for necessary information

[1353] Program processing explanation

[1354] Uploading and receiving invoices

[1355] Processing content

[1356] A user uploads an image or PDF file of the invoice to the system using a terminal.

[1357] The terminal stores the uploaded invoice file in a temporary storage area and transmits it to the server.

[1358] The server stores the received bill file in a temporary storage area.

[1359] Invoice scanning and OCR processing

[1360] Processing content

[1361] The server scans the received invoice file using OCR technology.

[1362] The server extracts text data from the invoice using OCR processing and stores it in a temporary database.

[1363] Data analysis and item extraction using generative AI

[1364] Processing content

[1365] The server uses the generative AI model to analyze and extract each item of the invoice (e.g., invoice number, issue date, amount, issuer information, etc.) from the OCR results.

[1366] The server stores the analyzed and extracted data in a database as structured data.

[1367] Verification of eligible invoice issuers

[1368] Processing content

[1369] The server accesses a database of eligible bill issuers and verifies the extracted issuer information.

[1370] Based on the result of the comparison, the server determines whether or not the invoice is a qualified invoice.

[1371] The server updates the result of the determination with the structured data.

[1372] Digitization and storage

[1373] Processing content

[1374] The server then generates the final invoice data as electronic data and stores it on a cloud platform.

[1375] The server notifies the terminal that the saving is complete.

[1376] QA Support and Emotion Engine

[1377] Processing content

[1378] The user makes a query using the terminal.

[1379] The terminal transmits the inquiry to the server.

[1380] The server uses generative AI to analyze the inquiry and generate an appropriate response.

[1381] The server uses an emotion engine as part of the QA system to analyze the user's emotions.

[1382] The server adjusts the content and tone of the response based on the analyzed emotional data to generate the optimal response.

[1383] The server sends the generated answer to the terminal and provides it to the user.

[1384] The server stores the emotional data in a learning database and uses it to improve the response accuracy of the generation AI.

[1385] Specific examples

[1386] For example, when a business digitizes invoices and determines whether they are qualified invoices, the user uploads an image of the invoice to the system from their device. The server then extracts the invoice's text data using OCR processing and uses generation AI to analyze and extract each item on the invoice. The extracted issuer information is compared with a database of qualified invoice issuers, and the invoice is determined to be qualified. Finally, the digitized invoice data is securely stored on a cloud platform, allowing users to access it at any time.

[1387] Additionally, when a user makes an inquiry, the server uses generative AI and an emotion engine to analyze the user's emotions and generate an appropriate response. For example, if the user is feeling anxious or angry, providing a polite response that reflects those emotions can improve user satisfaction. At the same time, this emotional data is stored in a learning database and used to improve the accuracy of future response generation.

[1388] The above is an embodiment of the present invention. This system not only automates bill processing but also improves the user experience.

[1389] The processing flow will be explained below.

[1390] Step 1:

[1391] A user uploads an image or PDF file of the invoice to the system using a terminal.

[1392] Step 2:

[1393] The terminal stores the uploaded invoice file in a temporary storage area and transmits it to the server.

[1394] Step 3:

[1395] The server passes the received invoice file to the OCR processing engine and begins extracting character data.

[1396] Step 4:

[1397] The server retrieves the character data generated by the OCR processing engine and stores it in a temporary text database.

[1398] Step 5:

[1399] The server launches the generation AI, analyzes the saved text data, and extracts each item of the invoice (invoice number, issue date, amount, issuer information, etc.).

[1400] Step 6:

[1401] The server stores each extracted item in a database as structured data.

[1402] Step 7:

[1403] The server accesses a database of eligible bill issuers and verifies the extracted issuer information.

[1404] Step 8:

[1405] Based on the result of the check, the server determines whether the bill is a qualified bill.

[1406] Step 9:

[1407] The server stores the judgment results together with the structured data as electronic data on a cloud platform.

[1408] Step 10:

[1409] The server notifies the terminal that the saving has been completed, and the user is also notified.

[1410] Step 11:

[1411] When a user makes an inquiry using a terminal, the inquiry content is transmitted from the terminal to the server.

[1412] Step 12:

[1413] The server uses generative AI to analyze the inquiry and generate an appropriate response.

[1414] Step 13:

[1415] The server uses an emotion engine as part of the QA system to analyze the user's emotions.

[1416] Step 14:

[1417] The server adjusts the content and tone of the response based on the analyzed emotional data to generate the optimal response.

[1418] Step 15:

[1419] The server sends the generated answer to the terminal and provides it to the user.

[1420] Step 16:

[1421] The server stores the emotional data in a learning database and uses it to improve the response accuracy of the generation AI.

[1422] Step 17:

[1423] The server performs daily backups of all data stored on the cloud platform to ensure data safety.

[1424] Step 18:

[1425] The server performs access control based on user privileges and maintains appropriate security for stored data.

[1426] The above is the specific processing flow in the system of the present invention, which automates invoice processing, significantly reducing the workload of business operators and improving the user experience.

[1427] Example 2

[1428] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1429] Conventional invoice processing systems have the problem of requiring a great deal of manual work and time for scanning, analyzing, judging, digitally storing invoices, and responding to user inquiries. Furthermore, they can only provide a uniform response to user inquiries, making it difficult to improve user satisfaction. To solve these problems, there is a need for a system that can fully automate invoice processing and generate appropriate responses based on the user's emotions.

[1430] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1431] In this invention, the server includes means for receiving an image or electronic file of an invoice, means for converting the received file into character data using optical character recognition technology, means for analyzing the converted character data using a generative AI model and extracting each item of the invoice, means for comparing the extracted invoice issuer information with a database of qualified invoice issuers, means for determining whether the invoice is a qualified invoice based on the comparison results, means for storing the determined invoice as electronic data on a cloud platform, means for generating responses to inquiries from users using the generative AI model, means for using an emotion analysis engine to analyze user emotions and adjust the content and tone of the response, and means for saving the emotion analysis data in a learning database and improving the response accuracy of the generative AI model. This enables full automation of invoice processing operations and the generation of appropriate responses according to user emotions.

[1432] 1. "Invoice" means a document used to request payment for goods or services.

[1433] 2. "Image" means any form in which visual data can be stored and displayed electronically.

[1434] 3. "Electronic files" refers to documents and data stored and managed in digital form.

[1435] 4. "Receiving" refers to obtaining data or information from an external source.

[1436] 5. "Optical character recognition technology" is a technology that analyzes letters and numbers in an image and converts them into text data.

[1437] 6. “Generative AI model” means a model that uses artificial intelligence to generate and analyze data.

[1438] 7. "Analysis" refers to examining data in detail to clarify its content and structure.

[1439] 8. "Item" means an individual element that refers to a specific piece of information or data.

[1440] 9. "Qualified Bill Issuer Database" is a database for verifying the eligibility of bill issuers.

[1441] 10. "Matching" refers to comparing different data to determine whether they match or fit.

[1442] 11. "Judgment" means determining suitability based on criteria.

[1443] 12. "Electronic data" means data stored and managed in digital form.

[1444] 13. "Cloud infrastructure" refers to infrastructure with data storage and processing capabilities provided via the Internet.

[1445] 14. "Inquiry" means a question or request for information or an answer.

[1446] 15. "Response" means an answer or reaction provided to an inquiry or request.

[1447] 16. "Sentiment analysis engine" refers to software technology for analyzing emotions from data such as text and voice.

[1448] 17. "Training Database" means a database that stores and manages data used to train machine learning models.

[1449] This invention is a system for streamlining invoice-related tasks, and it works in cooperation with the server, terminal, and user. Specifically, it is equipped with technology for scanning invoices, determining whether they are qualified invoices, digitizing them, storing them, responding to inquiries, and recognizing user emotions to optimize responses.

[1450] System configuration

[1451] Hardware and Software Configuration

[1452] Server: Use a high-performance cloud server (e.g., AWS EC2).

[1453] Device: The computer or smartphone used by the user.

[1454] OCR technology: Uses Google Cloud Vision API.

[1455] Generative AI model: OpenAI GPT-4 is used.

[1456] Cloud infrastructure: AWS S3 is used for data storage.

[1457] Sentiment analysis engine: Uses Microsoft Azure Emotion API.

[1458] Specific explanation of program processing

[1459] Uploading and receiving invoices

[1460] Users upload invoice images or PDF files to the system using a dedicated web interface on their device or a smartphone application. The device temporarily stores the uploaded file in local storage and then sends it to the server using HTTPS. The server then stores the file in a temporary storage area (AWS S3 bucket).

[1461] Invoice scanning and OCR processing

[1462] The server scans the received invoice file using Google Cloud Vision API and extracts the text data, which is then stored in a temporary database (MongoDB).

[1463] Data analysis and item extraction using generative AI

[1464] The server uses OpenAI GPT-4 to analyze and extract each item of the invoice (e.g., invoice number, issue date, amount, issuer information, etc.) from the OCR processing results. The analysis is performed by inputting the following prompt sentence into the generation AI:

[1465] Extract the invoice items from this text.

[1466] The parsed and extracted data is stored in a database (MySQL) as structured data (e.g., database records).

[1467] Verification of eligible invoice issuers

[1468] The server accesses the database of qualified invoice issuers (a database of the National Tax Agency) and compares the issuer information extracted by the generation AI. Based on the comparison results, the server determines that the invoice is a qualified invoice and updates the internal database.

[1469] Digitization and storage

[1470] The server generates the final invoice data as electronic data (e.g. PDF format) and saves it on the cloud infrastructure (AWS S3). It notifies the terminal that saving is complete.

[1471] QA Support and Emotion Engine

[1472] The user enters a question into a dedicated inquiry form and presses the send button. The device sends the information to the server, which uses OpenAI GPT-4 to analyze the inquiry and generate an appropriate answer. The server also analyzes the user's emotions using the Microsoft Azure Emotion API and adjusts the content and tone of the response. For example, the following prompt sentences can be used:

[1473] I have a question regarding invoice number 12345. The invoice was determined to be ineligible. I would like to know the reason for this and how I can resubmit it as an eligible invoice.

[1474] The final answer generated is provided to the user, and the sentiment analysis data is stored in a learning database to help improve the accuracy of future response generation.

[1475] Specific examples

[1476] For example, when a business digitizes invoices and determines whether they are qualified invoices, the user uploads an image of the invoice to the system from their device. The server uses OCR processing to extract the text data from the invoice, and uses generation AI to analyze and extract each item on the invoice. The extracted issuer information is compared with a database of qualified invoice issuers, and the invoice is determined to be qualified. The digitized invoice data is securely stored on a cloud platform, allowing users to access it at any time.

[1477] Additionally, when a user makes an inquiry, the server uses generative AI and an emotion engine to analyze the user's emotions and generate an appropriate response. For example, if a user is feeling anxious or angry, providing a polite response that reflects those emotions can improve user satisfaction. At the same time, this emotional data is stored in a learning database and used to improve the accuracy of future response generation. This system significantly improves the efficiency of invoice processing and the user experience.

[1478] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1479] Step 1: Upload your invoice

[1480] Users upload invoice images or PDF files to the system using a dedicated web interface or smartphone app. The input is the invoice image file or PDF file, and the output is saving it to the terminal's temporary storage area. The terminal temporarily saves the uploaded file in local storage and sends it to the server using the HTTPS protocol.

[1481] Step 2: Receiving and temporarily saving the file

[1482] The server receives the invoice file sent from the terminal and saves it in a temporary storage area within the cloud infrastructure (e.g. AWS S3). The input is the invoice file sent from the terminal, and the output is the file saved in the temporary storage area within the cloud infrastructure. The file is checked for integrity and scanned for viruses, and if there are no problems, it proceeds to the next processing step.

[1483] Step 3: OCR the invoice

[1484] The server scans the invoice file stored in the temporary storage area of ​​the cloud infrastructure using the Google Cloud Vision API. The input is the invoice file stored in the temporary storage area, and the output is text data. Specifically, the letters and numbers in the image are extracted as text data, and the data is stored in a temporary database (e.g., MongoDB).

[1485] Step 4: Data analysis with generative AI

[1486] The server uses OpenAI GPT-4 to analyze and extract each item of the invoice based on the extracted character data. The input is the extracted character data, and the output is structured data including each item of the invoice (e.g., invoice number, issue date, amount, issuer information, etc.). The following prompt sentence is input to the generation AI:

[1487] Extract the invoice items from this text.

[1488] The generated structured data is stored in a database (e.g., MySQL).

[1489] Step 5: Verify eligible billing entities

[1490] The server compares the issuer information stored in the database with the database of qualified invoice issuers (a database of the National Tax Agency). The input is the issuer information, and the output is the eligibility determination result. Specifically, the server compares the issuer information with the database and determines whether the invoice is appropriate based on the results.

[1491] Step 6: Save the relevance data

[1492] The server updates the database with structured data that reflects the judgment result. The input is the eligibility judgment result, and the output is the updated structured data.

[1493] Step 7: Electronic data generation and final storage

[1494] The server generates the final invoice data that has been determined to be eligible as electronic data and stores it on a cloud platform (e.g., AWS S3). The input is the determined invoice data, and the output is the final electronic data. A notification that the data has been saved is then sent to the terminal.

[1495] Step 8: Receiving inquiries from users

[1496] The user enters a question into a dedicated inquiry form and presses the send button. The input is the inquiry content, and the output is the transmission of the inquiry content to the server. The terminal then sends the inquiry content to the server.

[1497] Step 9: Parsing the query and generating a response

[1498] The server analyzes the query using OpenAI GPT-4 and generates an appropriate answer. The input is the query, and the output is the generated answer. It also analyzes the user's emotions using Microsoft Azure Emotion API and adjusts the content and tone of the response. For example, it uses the following prompt:

[1499] I have a question regarding invoice number 12345. The invoice was determined to be ineligible. I would like to know the reason for this and how I can resubmit it as an eligible invoice.

[1500] Step 10: Storing emotion data and serving responses

[1501] The server stores the generated answer together with the emotion analysis data in a learning database and transmits the answer to the terminal. The input is the generated answer and the emotion analysis data, and the output is saving the answer in the learning database and providing the answer to the user.

[1502] The above is the specific flow of operations in the system's processing steps. This system will greatly improve the efficiency of invoice processing and the user experience.

[1503] (Application example 2)

[1504] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1505] Improving customer service efficiency and customer satisfaction are currently key challenges for many brick-and-mortar stores. In particular, quick and accurate responses are required when handling inquiries and troubleshooting, which directly impacts the customer experience. However, handling a large number of inquiries requires human intervention, making it difficult to secure and manage a workforce. Furthermore, while properly understanding customer emotions and responding accordingly could contribute to further improving customer satisfaction, automation remains lacking in this area.

[1506] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for receiving an image or electronic document of an invoice; means for converting the received document into character data using optical character recognition technology; means for analyzing the converted character data using generative artificial intelligence and extracting each item of the invoice; means for comparing the extracted invoice issuer information with a database of qualified invoice issuers; means for determining whether the invoice is a qualified invoice based on the comparison result; means for storing the determined invoice as electronic data on a remote platform; means for generating a response to a user inquiry using generative artificial intelligence; and means for adjusting the response based on the emotion data using an emotion recognition engine that analyzes the user's emotions. This makes it possible for brick-and-mortar stores to automatically provide optimal responses based on the analysis of receipt and invoice information and the customer's emotions.

[1507] "Invoice image or electronic document" refers to an image file or a digital document such as a PDF that visually represents the contents of the invoice.

[1508] "Optical character recognition technology" is a technology for analyzing character information contained in an image and converting it into text data.

[1509] "Generative artificial intelligence" is an artificial intelligence technology that generates new data and responses based on large amounts of data, and is used for natural language generation and various data analyses.

[1510] The "Qualified Invoice Issuer Database" is a database that compiles information on businesses that have the authority to issue qualified invoices.

[1511] "Remote infrastructure" refers to cloud environments and remote data storage accessible via the Internet.

[1512] An "emotion recognition engine" is an engine that analyzes and identifies a user's emotional state from data such as text and voice.

[1513] "User emotion" refers to the user's psychological state and emotional expression during an inquiry or conversation.

[1514] "Response" or "response generation" is the process of automatically creating an appropriate reply to a user's inquiry using generative artificial intelligence.

[1515] System Overview

[1516] The system of the present invention includes a series of processes, from reading invoices to determining whether they are eligible, converting them into electronic data, storing them, responding to inquiries, and recognizing the user's emotions to generate the most appropriate response. This system works in cooperation with the server, terminals, and users, and makes full use of a variety of technologies, including OCR technology, generative artificial intelligence, and an emotion recognition engine.

[1517] Server Features

[1518] The server has the following main functions:

[1519] 1. Receiving invoices: Receiving invoice images and electronic documents uploaded by users.

[1520] 2. Optical character recognition processing: The received document is converted into character data using optical character recognition technology (such as pytesseract).

[1521] 3. Data analysis and item extraction: Using generative artificial intelligence (such as Hugging Face's transformers library), each item of the invoice (e.g., invoice number, issue date, amount, issuer information, etc.) is analyzed and extracted from the converted text data.

[1522] 4. Database matching: The extracted invoice issuer information is matched with the database of eligible invoice issuers to determine whether the invoice is eligible.

[1523] 5. Digitalization and storage: The judged invoices are stored as electronic data on a cloud platform.

[1524] 6. Inquiry handling and response generation: Generative artificial intelligence is used to generate appropriate responses to user inquiries.

[1525] 7. Sentiment Analysis: An emotion recognition engine is used to analyze the user's emotional data and tailor the response.

[1526] Device Features

[1527] The terminal provides the following functions to the user:

[1528] 1. Provide an interface: Through the user interface, you can upload invoices, send and receive inquiries, etc.

[1529] 2. Sending and receiving data: Two-way communication with the server to send or receive data.

[1530] User operations

[1531] The user does the following:

[1532] 1. Uploading invoices: Use the terminal to upload an image or electronic document of the invoice to the system.

[1533] 2. Sending a query: Use the terminal to query the system for the required information.

[1534] 3. Check the matching result: Check the matching result sent from the server.

[1535] Hardware and software used

[1536] The hardware used is a customer-facing robot from a brick-and-mortar store (e.g., a typical customer-facing robot), while the software uses OCR technology (pytesseract), generative artificial intelligence (Hugging Face's transformers library and GPT-2 model), and a sentiment analysis module (sentiment_analysis).

[1537] Specific examples

[1538] For example, if a customer asks "Can I get a refund for this item?" in a store, the robot will analyze the customer's facial expressions and tone of voice to determine their emotions, and if the emotion score is low, it will generate a more polite and flexible response. At the same time, if the customer shows the robot a receipt, it will use OCR technology to quickly read the information on the receipt and immediately determine whether a refund is possible and respond accordingly.

[1539] Prompt Sentence Examples

[1540] User Question: "Can I get a refund for this item?"

[1541] Depending on the sentiment analysis results, the generated response may be:

[1542] "Of course, I'm happy to refund this item." (High sentiment score)

[1543] "Sorry, but we need to go through some checks before we can process your refund. Could you please wait a moment?" (if your sentiment score is low)

[1544] This system is expected to improve the efficiency of customer service in physical stores and increase customer satisfaction.

[1545] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1546] Step 1:

[1547] The server receives the invoice image or electronic document uploaded by the user. The input format can be an image file, a PDF file, or other format. The received file is stored in a temporary storage area.

[1548] Step 2:

[1549] The server converts the received invoice into text data using optical character recognition (OCR). It scans the image or PDF file of the invoice received as input and generates text data as output. Specifically, it uses pytesseract to analyze the characters in the image or PDF and convert them into text format.

[1550] Step 3:

[1551] The server uses generative artificial intelligence to analyze and extract each item of the invoice (e.g., invoice number, issue date, amount, issuer information, etc.) from the converted character data. It uses text data generated by OCR as input and generates structured data containing each item of the invoice as output. Specifically, it uses a generative artificial intelligence model using Hugging Face's transformers library.

[1552] Step 4:

[1553] The server compares the extracted invoice issuer information with the database of qualified invoice issuers. It uses the issuer information and database information as input and obtains the matching result as output. This completes the process of determining whether the invoice is qualified.

[1554] Step 5:

[1555] Based on the judgment results, the server stores qualified invoices as electronic data on a cloud platform. Using the matching results and invoice data as input, it generates a message indicating that the data has been saved as output. The electronic data is securely stored in cloud storage and can be accessed by users.

[1556] Step 6:

[1557] When a user makes a query using a terminal, the server receives the query content. At this time, the query content is sent as input and stored on the server side for processing.

[1558] Step 7:

[1559] The server uses generative artificial intelligence to generate appropriate responses to user queries. It uses the user's query as input and obtains the generated response text as output. Specifically, it uses Hugging Face's transformers library to generate the optimal response corresponding to the query.

[1560] Step 8:

[1561] The server uses an emotion recognition engine to analyze the user's emotion data. It uses the query content, the user's tone of voice, and text data as input, and generates an emotion score as output. It uses a sentiment analysis module (such as Sentiment Analysis).

[1562] Step 9:

[1563] The server adjusts the content and tone of the response based on the analyzed emotion data. It uses the emotion score and the initial response text as input and generates an optimized response text as output, which provides an appropriate response that corresponds to the user's emotion.

[1564] Step 10:

[1565] The server sends the generated response to the terminal and provides it to the user, using the optimized response text as input and displaying it on the terminal as output, after which the user can check the generated response.

[1566] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1567] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1568] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1569] [Fourth embodiment]

[1570] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1571] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1572] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1573] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1574] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1575] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1576] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1577] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1578] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1579] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1580] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1581] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1582] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1583] An embodiment of the present invention will be described below as a specific example.

[1584] System Overview

[1585] This invention is a system that automates everything from reading invoices to determining whether they are qualified invoices, digitizing them, storing them, and responding to inquiries, in order to streamline invoice-related operations for businesses. This system works in cooperation with three parties: a server, a terminal, and a user.

[1586] System configuration

[1587] server

[1588] Receiving invoice data

[1589] Text extraction using OCR processing

[1590] Data analysis and item extraction using generative AI

[1591] Matching with the database of eligible bill issuers

[1592] Electronic data storage based on the judgment results

[1593] Responding to inquiries through the QA system

[1594] Terminal

[1595] Providing a user interface

[1596] Sending and receiving data to the server

[1597] User

[1598] Uploading an invoice

[1599] Checking the matching results

[1600] Inquiry for necessary information

[1601] Program processing explanation

[1602] Uploading and receiving invoices

[1603] Processing content

[1604] A user uploads an image or PDF file of the invoice to the system using a terminal.

[1605] The terminal sends the uploaded bill file to the server.

[1606] The server stores the received bill file in a temporary storage area.

[1607] Invoice scanning and OCR processing

[1608] Processing content

[1609] The server scans the received invoice file using OCR technology.

[1610] The server extracts text data from the invoice using OCR processing and stores it in a temporary database.

[1611] Data analysis and item extraction using generative AI

[1612] Processing content

[1613] The server uses the generative AI model to analyze and extract each item of the invoice (e.g., invoice number, issue date, amount, issuer information, etc.) from the OCR results.

[1614] The server stores the analyzed and extracted data in a database as structured data.

[1615] Verification of eligible invoice issuers

[1616] Processing content

[1617] The server accesses a database of eligible bill issuers and verifies the extracted issuer information.

[1618] Based on the result of the comparison, the server determines whether or not the invoice is a qualified invoice.

[1619] The server updates the result of the determination with the structured data.

[1620] Digitization and storage

[1621] Processing content

[1622] The server then generates the final invoice data as electronic data and stores it on a cloud platform.

[1623] The server notifies the terminal that the saving is complete.

[1624] QA Support

[1625] Processing content

[1626] The user makes a query using the terminal.

[1627] The terminal transmits the inquiry to the server.

[1628] The server uses generative AI to analyze the inquiry and generate an appropriate response.

[1629] The server sends the generated answer to the terminal and provides it to the user.

[1630] Specific examples

[1631] For example, when a business digitizes invoices and determines whether they are qualified invoices, the user uploads an image of the invoice to the system from their device. The server then extracts the invoice's text data using OCR processing, and uses generation AI to analyze and extract each item on the invoice. The extracted issuer information is compared with a database of qualified invoice issuers, and the invoice is determined to be qualified. Finally, the digitized invoice data is securely stored on a cloud platform, allowing users to access it at any time. Furthermore, if the user makes an inquiry, the generation AI can respond quickly.

[1632] The above is an embodiment of the present invention. This system automates invoice processing and significantly improves operational efficiency.

[1633] The processing flow will be explained below.

[1634] Step 1:

[1635] A user uploads an image or PDF file of the invoice to the system using a terminal.

[1636] Step 2:

[1637] The terminal stores the uploaded invoice file in a temporary storage area and transmits it to the server.

[1638] Step 3:

[1639] The server passes the received invoice file to the OCR processing engine and begins extracting character data.

[1640] Step 4:

[1641] The server retrieves the character data generated by the OCR processing engine and stores it in a temporary text database.

[1642] Step 5:

[1643] The server launches the generation AI, analyzes the saved text data, and extracts each item of the invoice (invoice number, issue date, amount, issuer information, etc.).

[1644] Step 6:

[1645] The server stores each extracted item in a database as structured data.

[1646] Step 7:

[1647] The server accesses a database of eligible bill issuers and verifies the extracted issuer information.

[1648] Step 8:

[1649] Based on the result of the check, the server determines whether the bill is a qualified bill.

[1650] Step 9:

[1651] The server stores the judgment results together with the structured data as electronic data on a cloud platform.

[1652] Step 10:

[1653] The server notifies the terminal that the saving has been completed, and the user is also notified.

[1654] Step 11:

[1655] When a user makes an inquiry using a terminal, the inquiry content is transmitted from the terminal to the server.

[1656] Step 12:

[1657] The server uses generative AI to analyze the inquiry and generate an appropriate response.

[1658] Step 13:

[1659] The server sends the generated answer to the terminal and provides it to the user.

[1660] Step 14:

[1661] The server performs daily backups of all data stored on the cloud platform to ensure data safety.

[1662] Step 15:

[1663] The server performs access control based on user privileges and maintains appropriate security for stored data.

[1664] The above is the specific processing flow in the system of the present invention, which automates the invoice processing work and significantly reduces the workload of business operators.

[1665] Example 1

[1666] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1667] Traditional invoice processing operations require a lot of manual input and verification, which takes time and effort. There is also a high risk of human error, making it necessary to streamline operations. Furthermore, delays in determining and responding to qualified invoices can expose businesses to legal risks and a decline in their credibility. Furthermore, the lack of a system for quickly responding to user inquiries makes it difficult to provide reliable services.

[1668] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1669] In this invention, the server includes means for receiving an invoice document file, means for converting the received file into character data using optical character recognition technology, means for analyzing the converted character data using a generation AI and extracting each item of the invoice, means for comparing the extracted invoice issuer information with a qualified invoice issuer database, means for determining whether the invoice is a qualified invoice based on the comparison results, means for storing the determined invoice as electronic data on a cloud platform, and means for generating responses to user inquiries using a generation AI. This reduces manual work, improves business efficiency and accuracy, and enables early determination of qualified invoices and secure electronic storage. Furthermore, user inquiries can be responded to quickly and accurately.

[1670] An "invoice document file" is a file that electronically stores a document used as an invoice, and refers to a digital file in image format, PDF format, or the like.

[1671] "Optical character recognition technology" is a technology that analyzes and extracts character information from images or documents containing handwritten characters, and is also known as OCR (Optical Character Recognition).

[1672] "Generative AI" is a technology that uses machine learning models and artificial intelligence to analyze and generate data, and is particularly used in natural language processing and image analysis.

[1673] The "Qualified Invoice Issuer Database" is a database for registering and managing information on businesses that issue qualified invoices, which allows you to check whether a specific business issues qualified invoices.

[1674] A "cloud platform" is a foundation for using infrastructure such as computing resources and storage provided via the Internet, and is provided as a cloud service.

[1675] "User inquiries" refer to requests made by system users to the system for information or to resolve questions.

[1676] This invention is a system designed to help businesses streamline invoice-related operations. This system functions through the cooperation of three parties: the server, the terminal, and the user. Specifically, it automates everything from uploading invoices, OCR processing, data analysis using AI generation, and determining whether an invoice is eligible, to digitizing and saving the data, and responding to inquiries.

[1677] System configuration

[1678] The system is divided into the following main components:

[1679] server

[1680] The server performs many functions, including:

[1681] 1. Receiving billing data: Receives the billing file sent from the user or terminal.

[1682] 2. OCR processing: Extract character data from the received invoice file using optical character recognition technology (e.g., Google Cloud Vision API or Tesseract OCR).

[1683] 3. Data analysis using generative AI: A generative AI model (e.g., OpenAI GPT-4) is used to parse and extract each item of the invoice from the OCR results.

[1684] 4. Matching with the database of qualified invoice issuers: The extracted issuer information is matched with the database to determine whether the invoice is a qualified invoice.

[1685] 5. Electronic data storage: The adjudicated invoice data is stored electronically on a cloud platform (e.g., AWS S3).

[1686] 6. Response to inquiries using a QA system: Using generative AI, we analyze user inquiries and generate appropriate responses.

[1687] Terminal

[1688] The terminal provides the following features:

[1689] 1. User Interface: Provides an interface for users to upload invoices, check matching results, and make inquiries.

[1690] 2. Data transmission and reception: Sends billing files and inquiry details to the server and receives responses from the server.

[1691] User

[1692] The user does the following:

[1693] 1. Upload invoice: Upload an image or PDF file of your invoice using our dedicated application or web interface.

[1694] 2. Check the matching results: Check the matching results generated by the server and the electronic invoice data.

[1695] 3. Inquiry: Submitting a request to the system for information or to resolve a question.

[1696] Specific examples

[1697] For example, if a business wants to digitize invoices and determine which invoices are eligible, it would follow the steps below:

[1698] 1. The user uses a terminal to upload an image of the invoice to the system. Specifically, typically, the user takes a photo of the invoice using the camera function of a smartphone and uploads the image file.

[1699] 2. The terminal sends the bill file to the server.

[1700] 3. The server receives the file and extracts text data from the invoice using OCR processing, using technologies such as Google Cloud Vision API and Tesseract OCR.

[1701] 4. The server uses a generative AI model (such as OpenAI GPT-4) to analyze and extract each item of the invoice (invoice number, issue date, amount, issuer information, etc.) from the OCR results.

[1702] 5. The server checks the issuer information against the database of qualified invoice issuers and determines that the invoice is a qualified invoice.

[1703] 6. The server electronically stores the adjudicated invoice data on a cloud platform (e.g., AWS S3).

[1704] 7. Users can access the stored data to check the matching results, and if they make inquiries, they can receive a prompt and appropriate response from the generating AI.

[1705] Prompt Sentence Examples

[1706] An example of an input to a generative AI model might be a prompt like this:

[1707] "A user has used their device to upload an image of an invoice. What are the next steps?"

[1708] By using this system, invoice processing operations can be automated, improving work efficiency and accuracy. Furthermore, it becomes possible to respond to user inquiries quickly and accurately.

[1709] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1710] Step 1: Upload your invoice

[1711] The user uploads an image or PDF file of the invoice to the system using the terminal. Specifically, the user clicks the "File Upload" button in the dedicated application or web interface, selects the relevant invoice file, and displays it.

[1712] Input: Invoice image or PDF file.

[1713] Output: The upload request sent to the device.

[1714] Step 2: Submit your invoice file

[1715] The terminal sends the invoice file uploaded by the user to the server using an HTTP POST request, along with the file metadata and user ID.

[1716] Input: User selected invoice file and metadata.

[1717] Output: The file sent to the server along with any accompanying information.

[1718] Step 3: Save the file temporarily

[1719] The server stores the received invoice file in a temporary storage area (for example, an Amazon S3 bucket or local storage). The file name includes the user ID and a timestamp to make it uniquely identifiable.

[1720] Input: Invoice file sent from the terminal.

[1721] Output: Temporarily saved file.

[1722] Step 4: OCR processing

[1723] The server scans the invoice file using OCR technology (Google Cloud Vision API and Tesseract OCR) and extracts text data from the image. After OCR processing, the extracted text data is stored in a temporary database.

[1724] Input: Temporarily saved invoice file.

[1725] Output: The extracted text data.

[1726] Step 5: Temporarily save the text data

[1727] The server stores the text data obtained through the OCR process in a temporary database, and checks the data integrity in the process, specifically logging any incomplete or erroneous data in case it needs to be reprocessed.

[1728] Input: Text data extracted by OCR processing.

[1729] Output: Temporarily saved text data and processing log.

[1730] Step 6: Data analysis and item extraction using generative AI

[1731] The server uses a generative AI model (such as OpenAI GPT-4) to parse and extract each item of the invoice from the OCR results (invoice number, issue date, amount, issuer information, etc.), specifically identifying important entities and organizing them into appropriate fields in a database.

[1732] Input: Temporarily saved text data.

[1733] Output: Structured item data.

[1734] Step 7: Save the analysis data

[1735] The server stores the analyzed and extracted data in this database as structured data in JSON or SQL table format to facilitate subsequent query processing.

[1736] Input: Item data analyzed and extracted by the generative AI.

[1737] Output: Structured data stored in this database.

[1738] Step 8: Match eligible invoice issuers

[1739] The server accesses a database of eligible bill issuers and matches the extracted issuer information by searching the issuer information using a query to determine whether there is a match.

[1740] Input: Structured issuer information.

[1741] Output: Issuer match result.

[1742] Step 9: Update the results

[1743] Based on the result of the matching, the server determines whether the invoice is a qualified invoice or not, and updates the result along with the structured data, adding a determination flag and status information.

[1744] Input: Issuer match result.

[1745] Output: Updated decision result data.

[1746] Step 10: Digitization and storage

[1747] The server generates the final invoice data as electronic data and stores it securely on a cloud platform (such as AWS S3). The data is also encrypted when stored.

[1748] Input: Judgment result data.

[1749] Output: Electronic data stored on a cloud platform.

[1750] Step 11: Sending a save completion notification

[1751] The server generates a notification of completion of saving and sends it to the terminal. The notification includes the URL of the invoice data and status information.

[1752] Input: Saved invoice data.

[1753] Output: Notifications sent to the device.

[1754] Step 12: Submitting an inquiry

[1755] A user uses a terminal to make an inquiry to the system. Specifically, the user enters the inquiry details in the input form and clicks the send button.

[1756] Input: Enquiry details.

[1757] Output: The query request generated on the terminal.

[1758] Step 13: Sending the query request to the server

[1759] The terminal converts the user's inquiry into JSON format and sends it to the server via an HTTP POST request.

[1760] Input: User's inquiry.

[1761] Output: The request sent to the server.

[1762] Step 14: Generate and send a response using the generative AI

[1763] The server uses generative AI (such as OpenAI GPT-4) to analyze the query and generate an appropriate answer, which is then sent to the device and served to the user.

[1764] Input: User's inquiry.

[1765] Output: The response sent to the terminal.

[1766] This concludes the detailed explanation of each processing step of the system. By implementing this system, invoice processing operations can be automated and made more efficient.

[1767] (Application example 1)

[1768] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1769] In conventional logistics centers, product inspection work is done manually, which is inefficient and prone to human error. Furthermore, inspectors need a great deal of prior knowledge and experience to determine whether a product is suitable. Furthermore, responding to inquiries is also manual, which often results in a long response time. There is a need to solve these problems and improve the operational efficiency of logistics centers.

[1770] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1771] In this invention, the server includes a means for receiving an image file of an invoice or label, a means for converting the received file into character data using OCR technology, and a means for analyzing the converted character data using a generation AI to extract each item. This enables the automation of product inspection work at logistics centers and the determination of qualified products.

[1772] "Invoice or label image file" means a scanned image or digital file saved in PDF format of a label or invoice for a product handled at a distribution center or other operation.

[1773] "OCR technology" is an abbreviation for optical character recognition technology, which extracts character information such as letters and numbers from image files and converts them into digital data.

[1774] "Generative AI" is a type of artificial intelligence that uses machine learning algorithms to analyze large amounts of data and generate new information and answers.

[1775] The "eligibility determination database" is a database for collating extracted issuer information or product information to determine its eligibility.

[1776] "Cloud infrastructure" refers to cloud computing infrastructure provided via the Internet, and is a platform for storing, processing, and managing data.

[1777] An "enquiry" is a question or question posed by a user to the system for which the system will generate an answer.

[1778] System Overview

[1779] This invention is a system for automating product inspection work at logistics centers. The system uses smart glasses to scan product labels, analyzes the information in real time, determines whether the product is eligible, and responds to inquiries as needed. The information is then securely stored on a cloud platform.

[1780] System configuration

[1781] Smart Glasses

[1782] The product label is scanned and the image data is sent to the server.

[1783] server

[1784] The received image data is converted into character data using OCR technology.

[1785] The generated text data is analyzed using generative AI to extract each item.

[1786] The extracted product information is compared with the eligibility assessment database, and the assessment results are stored on a cloud platform.

[1787] Generative AI is used to generate responses to user inquiries.

[1788] Cloud infrastructure

[1789] The determined data is stored securely and provided so that the user can access it.

[1790] Specific implementation steps

[1791] 1. Initiate inspection and scan:

[1792] A user scans a product label using smart glasses.

[1793] The smart glasses send the scanned image data to a server.

[1794] 2. Data extraction using OCR processing:

[1795] The server converts the received image data into text data using OCR technology (for example, Google Cloud Vision API).

[1796] The converted character data is temporarily saved.

[1797] 3. Data analysis and eligibility judgment by generative AI:

[1798] The server uses a generative AI model (e.g., OpenAI GPT model) to analyze the text data and extract each item, such as product code, quantity, and expiration date.

[1799] The extracted product information is compared with an eligibility determination database to determine the product's eligibility.

[1800] The determined data is stored on a cloud platform (e.g., AWS S3).

[1801] 4. QA Support:

[1802] When a user makes a query, the server uses generative AI to analyze the query and generate an appropriate answer.

[1803] The server provides the generated answer to the user.

[1804] Specific examples

[1805] Warehouse workers at the distribution center use smart glasses to scan the labels on incoming products. The server analyzes the data in real time and immediately determines whether the product is eligible. If it is eligible, the evidence is stored in AWS S3. If it is not eligible, the system notifies the user of the appropriate measures. Inquiries are also handled promptly using the generative AI model.

[1806] Example prompts to input to a generative AI model:

[1807] "Analyze the information on the product label scanned by the smart glasses, extract data such as the product code, quantity, and expiration date, and then use that information to determine whether the product is eligible. Also, generate an appropriate response to inquiries from users. If the product is eligible, store the data in a cloud platform and notify the user."

[1808] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1809] Step 1:

[1810] A user scans a product label using smart glasses.

[1811] Input: A user puts on the smart glasses and captures an image of a product label.

[1812] Output: The captured image data.

[1813] What it does: When the smart glasses see a product label, they activate their camera and scan the image of the label.

[1814] Step 2:

[1815] The smart glasses send the scanned image data to a server.

[1816] Input: Scanned product label image.

[1817] Output: Image data sent to the server.

[1818] How it works: The smart glasses upload the label image data to the server via a wireless network.

[1819] Step 3:

[1820] The server converts the received image data into text data using OCR technology.

[1821] Input: Received label image data.

[1822] Output: Character data extracted by OCR technology.

[1823] What it does: The server uses an OCR service such as Google Cloud Vision API to extract text information from the label image.

[1824] Step 4:

[1825] The server uses a generative AI model to analyze the converted text data and extract each item.

[1826] Input: Character data obtained by OCR processing.

[1827] Output: Information about each item, such as product code, quantity, and expiration date.

[1828] How it works: The server uses the OpenAI GPT model to analyze the text data and identify and extract the required data items.

[1829] Step 5:

[1830] The server compares the extracted product information with an eligibility determination database to determine eligibility.

[1831] Input: Extracted product information (product code, quantity, expiration date, etc.).

[1832] Output: Product eligibility determination result.

[1833] What happens: The server checks the product information against the eligibility database to determine if the product meets the criteria.

[1834] Step 6:

[1835] The server stores the determined data on a cloud platform.

[1836] Input: Data for the determined eligible product.

[1837] Output: Eligible data stored in a cloud infrastructure.

[1838] What it does: The server stores the data in cloud storage such as AWS S3, making it available for users to access later.

[1839] Step 7:

[1840] The server uses a generation AI to generate responses to user inquiries.

[1841] Input: User's inquiry.

[1842] Output: The generated response to the query.

[1843] Specific operation: Using a generative AI model (e.g., OpenAI GPT model), the query content is analyzed, an optimal answer is generated, and provided to the user.

[1844] In this way, by carrying out specific processing at each step, product inspection work at the logistics center is automated efficiently and accurately.

[1845] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1846] An embodiment of the present invention will be described below as a specific example.

[1847] System Overview

[1848] This invention is a system that allows businesses to improve the efficiency of their invoice-related operations by scanning invoices, determining whether they are qualified, digitizing them, storing them, responding to inquiries, and recognizing user emotions to optimize responses. This system works in cooperation with the server, terminals, and users, and makes use of a variety of technologies.

[1849] System configuration

[1850] server

[1851] Receiving invoice data

[1852] Text extraction using OCR processing

[1853] Data analysis and item extraction using generative AI

[1854] Matching with the database of eligible bill issuers

[1855] Electronic data storage based on the judgment results

[1856] Responding to inquiries through the QA system

[1857] Emotional analysis of users using an emotion engine

[1858] Response optimization based on emotional data

[1859] Learning database updates

[1860] Terminal

[1861] Providing a user interface

[1862] Sending and receiving data to the server

[1863] User

[1864] Uploading an invoice

[1865] Checking the matching results

[1866] Inquiry for necessary information

[1867] Program processing explanation

[1868] Uploading and receiving invoices

[1869] Processing content

[1870] A user uploads an image or PDF file of the invoice to the system using a terminal.

[1871] The terminal stores the uploaded invoice file in a temporary storage area and transmits it to the server.

[1872] The server stores the received bill file in a temporary storage area.

[1873] Invoice scanning and OCR processing

[1874] Processing content

[1875] The server scans the received invoice file using OCR technology.

[1876] The server extracts text data from the invoice using OCR processing and stores it in a temporary database.

[1877] Data analysis and item extraction using generative AI

[1878] Processing content

[1879] The server uses the generative AI model to analyze and extract each item of the invoice (e.g., invoice number, issue date, amount, issuer information, etc.) from the OCR results.

[1880] The server stores the analyzed and extracted data in a database as structured data.

[1881] Verification of eligible invoice issuers

[1882] Processing content

[1883] The server accesses a database of eligible bill issuers and verifies the extracted issuer information.

[1884] Based on the result of the comparison, the server determines whether or not the invoice is a qualified invoice.

[1885] The server updates the result of the determination with the structured data.

[1886] Digitization and storage

[1887] Processing content

[1888] The server then generates the final invoice data as electronic data and stores it on a cloud platform.

[1889] The server notifies the terminal that the saving is complete.

[1890] QA Support and Emotion Engine

[1891] Processing content

[1892] The user makes a query using the terminal.

[1893] The terminal transmits the inquiry to the server.

[1894] The server uses generative AI to analyze the inquiry and generate an appropriate response.

[1895] The server uses an emotion engine as part of the QA system to analyze the user's emotions.

[1896] The server adjusts the content and tone of the response based on the analyzed emotional data to generate the optimal response.

[1897] The server sends the generated answer to the terminal and provides it to the user.

[1898] The server stores the emotional data in a learning database and uses it to improve the response accuracy of the generation AI.

[1899] Specific examples

[1900] For example, when a business digitizes invoices and determines whether they are qualified invoices, the user uploads an image of the invoice to the system from their device. The server then extracts the invoice's text data using OCR processing and uses generation AI to analyze and extract each item on the invoice. The extracted issuer information is compared with a database of qualified invoice issuers, and the invoice is determined to be qualified. Finally, the digitized invoice data is securely stored on a cloud platform, allowing users to access it at any time.

[1901] Additionally, when a user makes an inquiry, the server uses generative AI and an emotion engine to analyze the user's emotions and generate an appropriate response. For example, if the user is feeling anxious or angry, providing a polite response that reflects those emotions can improve user satisfaction. At the same time, this emotional data is stored in a learning database and used to improve the accuracy of future response generation.

[1902] The above is an embodiment of the present invention. This system not only automates bill processing but also improves the user experience.

[1903] The processing flow will be explained below.

[1904] Step 1:

[1905] A user uploads an image or PDF file of the invoice to the system using a terminal.

[1906] Step 2:

[1907] The terminal stores the uploaded invoice file in a temporary storage area and transmits it to the server.

[1908] Step 3:

[1909] The server passes the received invoice file to the OCR processing engine and begins extracting character data.

[1910] Step 4:

[1911] The server retrieves the character data generated by the OCR processing engine and stores it in a temporary text database.

[1912] Step 5:

[1913] The server launches the generation AI, analyzes the saved text data, and extracts each item of the invoice (invoice number, issue date, amount, issuer information, etc.).

[1914] Step 6:

[1915] The server stores each extracted item in a database as structured data.

[1916] Step 7:

[1917] The server accesses a database of eligible bill issuers and verifies the extracted issuer information.

[1918] Step 8:

[1919] Based on the result of the check, the server determines whether the bill is a qualified bill.

[1920] Step 9:

[1921] The server stores the judgment results together with the structured data as electronic data on a cloud platform.

[1922] Step 10:

[1923] The server notifies the terminal that the saving has been completed, and the user is also notified.

[1924] Step 11:

[1925] When a user makes an inquiry using a terminal, the inquiry content is transmitted from the terminal to the server.

[1926] Step 12:

[1927] The server uses generative AI to analyze the inquiry and generate an appropriate response.

[1928] Step 13:

[1929] The server uses an emotion engine as part of the QA system to analyze the user's emotions.

[1930] Step 14:

[1931] The server adjusts the content and tone of the response based on the analyzed emotional data to generate the optimal response.

[1932] Step 15:

[1933] The server sends the generated answer to the terminal and provides it to the user.

[1934] Step 16:

[1935] The server stores the emotional data in a learning database and uses it to improve the response accuracy of the generation AI.

[1936] Step 17:

[1937] The server performs daily backups of all data stored on the cloud platform to ensure data safety.

[1938] Step 18:

[1939] The server performs access control based on user privileges and maintains appropriate security for stored data.

[1940] The above is the specific processing flow in the system of the present invention, which automates invoice processing, significantly reducing the workload of business operators and improving the user experience.

[1941] Example 2

[1942] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1943] Conventional invoice processing systems have the problem of requiring a great deal of manual work and time for scanning, analyzing, judging, digitally storing invoices, and responding to user inquiries. Furthermore, they can only provide a uniform response to user inquiries, making it difficult to improve user satisfaction. To solve these problems, there is a need for a system that can fully automate invoice processing and generate appropriate responses based on the user's emotions.

[1944] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1945] In this invention, the server includes means for receiving an image or electronic file of an invoice, means for converting the received file into character data using optical character recognition technology, means for analyzing the converted character data using a generative AI model and extracting each item of the invoice, means for comparing the extracted invoice issuer information with a database of qualified invoice issuers, means for determining whether the invoice is a qualified invoice based on the comparison results, means for storing the determined invoice as electronic data on a cloud platform, means for generating responses to inquiries from users using the generative AI model, means for using an emotion analysis engine to analyze user emotions and adjust the content and tone of the response, and means for saving the emotion analysis data in a learning database and improving the response accuracy of the generative AI model. This enables full automation of invoice processing operations and the generation of appropriate responses according to user emotions.

[1946] 1. "Invoice" means a document used to request payment for goods or services.

[1947] 2. "Image" means any form in which visual data can be stored and displayed electronically.

[1948] 3. "Electronic files" refers to documents and data stored and managed in digital form.

[1949] 4. "Receiving" refers to obtaining data or information from an external source.

[1950] 5. "Optical character recognition technology" is a technology that analyzes letters and numbers in an image and converts them into text data.

[1951] 6. “Generative AI model” means a model that uses artificial intelligence to generate and analyze data.

[1952] 7. "Analysis" refers to examining data in detail to clarify its content and structure.

[1953] 8. "Item" means an individual element that refers to a specific piece of information or data.

[1954] 9. "Qualified Bill Issuer Database" is a database for verifying the eligibility of bill issuers.

[1955] 10. "Matching" refers to comparing different data to determine whether they match or fit.

[1956] 11. "Judgment" means determining suitability based on criteria.

[1957] 12. "Electronic data" means data stored and managed in digital form.

[1958] 13. "Cloud infrastructure" refers to infrastructure with data storage and processing capabilities provided via the Internet.

[1959] 14. "Inquiry" means a question or request for information or an answer.

[1960] 15. "Response" means an answer or reaction provided to an inquiry or request.

[1961] 16. "Sentiment analysis engine" refers to software technology for analyzing emotions from data such as text and voice.

[1962] 17. "Training Database" means a database that stores and manages data used to train machine learning models.

[1963] This invention is a system for streamlining invoice-related tasks, and it works in cooperation with the server, terminal, and user. Specifically, it is equipped with technology for scanning invoices, determining whether they are qualified invoices, digitizing them, storing them, responding to inquiries, and recognizing user emotions to optimize responses.

[1964] System configuration

[1965] Hardware and Software Configuration

[1966] Server: Use a high-performance cloud server (e.g., AWS EC2).

[1967] Device: The computer or smartphone used by the user.

[1968] OCR technology: Uses Google Cloud Vision API.

[1969] Generative AI model: OpenAI GPT-4 is used.

[1970] Cloud infrastructure: AWS S3 is used for data storage.

[1971] Sentiment analysis engine: Uses Microsoft Azure Emotion API.

[1972] Specific explanation of program processing

[1973] Uploading and receiving invoices

[1974] Users upload invoice images or PDF files to the system using a dedicated web interface on their device or a smartphone application. The device temporarily stores the uploaded file in local storage and then sends it to the server using HTTPS. The server then stores the file in a temporary storage area (AWS S3 bucket).

[1975] Invoice scanning and OCR processing

[1976] The server scans the received invoice file using Google Cloud Vision API and extracts the text data, which is then stored in a temporary database (MongoDB).

[1977] Data analysis and item extraction using generative AI

[1978] The server uses OpenAI GPT-4 to analyze and extract each item of the invoice (e.g., invoice number, issue date, amount, issuer information, etc.) from the OCR processing results. The analysis is performed by inputting the following prompt sentence into the generation AI:

[1979] Extract the invoice items from this text.

[1980] The parsed and extracted data is stored in a database (MySQL) as structured data (e.g., database records).

[1981] Verification of eligible invoice issuers

[1982] The server accesses the database of qualified invoice issuers (a database of the National Tax Agency) and compares the issuer information extracted by the generation AI. Based on the comparison results, the server determines that the invoice is a qualified invoice and updates the internal database.

[1983] Digitization and storage

[1984] The server generates the final invoice data as electronic data (e.g. PDF format) and saves it on the cloud infrastructure (AWS S3). It notifies the terminal that saving is complete.

[1985] QA Support and Emotion Engine

[1986] The user enters a question into a dedicated inquiry form and presses the send button. The device sends the information to the server, which uses OpenAI GPT-4 to analyze the inquiry and generate an appropriate answer. The server also analyzes the user's emotions using the Microsoft Azure Emotion API and adjusts the content and tone of the response. For example, the following prompt sentences can be used:

[1987] I have a question regarding invoice number 12345. The invoice was determined to be ineligible. I would like to know the reason for this and how I can resubmit it as an eligible invoice.

[1988] The final answer generated is provided to the user, and the sentiment analysis data is stored in a learning database to help improve the accuracy of future response generation.

[1989] Specific examples

[1990] For example, when a business digitizes invoices and determines whether they are qualified invoices, the user uploads an image of the invoice to the system from their device. The server uses OCR processing to extract the text data from the invoice, and uses generation AI to analyze and extract each item on the invoice. The extracted issuer information is compared with a database of qualified invoice issuers, and the invoice is determined to be qualified. The digitized invoice data is securely stored on a cloud platform, allowing users to access it at any time.

[1991] Additionally, when a user makes an inquiry, the server uses generative AI and an emotion engine to analyze the user's emotions and generate an appropriate response. For example, if a user is feeling anxious or angry, providing a polite response that reflects those emotions can improve user satisfaction. At the same time, this emotional data is stored in a learning database and used to improve the accuracy of future response generation. This system significantly improves the efficiency of invoice processing and the user experience.

[1992] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1993] Step 1: Upload your invoice

[1994] Users upload invoice images or PDF files to the system using a dedicated web interface or smartphone app. The input is the invoice image file or PDF file, and the output is saving it to the terminal's temporary storage area. The terminal temporarily saves the uploaded file in local storage and sends it to the server using the HTTPS protocol.

[1995] Step 2: Receiving and temporarily saving the file

[1996] The server receives the invoice file sent from the terminal and saves it in a temporary storage area within the cloud infrastructure (e.g. AWS S3). The input is the invoice file sent from the terminal, and the output is the file saved in the temporary storage area within the cloud infrastructure. The file is checked for integrity and scanned for viruses, and if there are no problems, it proceeds to the next processing step.

[1997] Step 3: OCR the invoice

[1998] The server scans the invoice file stored in the temporary storage area of ​​the cloud infrastructure using the Google Cloud Vision API. The input is the invoice file stored in the temporary storage area, and the output is text data. Specifically, the letters and numbers in the image are extracted as text data, and the data is stored in a temporary database (e.g., MongoDB).

[1999] Step 4: Data analysis with generative AI

[2000] The server uses OpenAI GPT-4 to analyze and extract each item of the invoice based on the extracted character data. The input is the extracted character data, and the output is structured data including each item of the invoice (e.g., invoice number, issue date, amount, issuer information, etc.). The following prompt sentence is input to the generation AI:

[2001] Extract the invoice items from this text.

[2002] The generated structured data is stored in a database (e.g., MySQL).

[2003] Step 5: Verify eligible billing entities

[2004] The server compares the issuer information stored in the database with the database of qualified invoice issuers (a database of the National Tax Agency). The input is the issuer information, and the output is the eligibility determination result. Specifically, the server compares the issuer information with the database and determines whether the invoice is appropriate based on the results.

[2005] Step 6: Save the relevance data

[2006] The server updates the database with structured data that reflects the judgment result. The input is the eligibility judgment result, and the output is the updated structured data.

[2007] Step 7: Electronic data generation and final storage

[2008] The server generates the final invoice data that has been determined to be eligible as electronic data and stores it on a cloud platform (e.g., AWS S3). The input is the determined invoice data, and the output is the final electronic data. A notification that the data has been saved is then sent to the terminal.

[2009] Step 8: Receiving inquiries from users

[2010] The user enters a question into a dedicated inquiry form and presses the send button. The input is the inquiry content, and the output is the transmission of the inquiry content to the server. The terminal then sends the inquiry content to the server.

[2011] Step 9: Parsing the query and generating a response

[2012] The server analyzes the query using OpenAI GPT-4 and generates an appropriate answer. The input is the query, and the output is the generated answer. It also analyzes the user's emotions using Microsoft Azure Emotion API and adjusts the content and tone of the response. For example, it uses the following prompt:

[2013] I have a question regarding invoice number 12345. The invoice was determined to be ineligible. I would like to know the reason for this and how I can resubmit it as an eligible invoice.

[2014] Step 10: Storing emotion data and serving responses

[2015] The server stores the generated answer together with the emotion analysis data in a learning database and transmits the answer to the terminal. The input is the generated answer and the emotion analysis data, and the output is saving the answer in the learning database and providing the answer to the user.

[2016] The above is the specific flow of operations in the system's processing steps. This system will greatly improve the efficiency of invoice processing and the user experience.

[2017] (Application example 2)

[2018] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2019] Improving customer service efficiency and customer satisfaction are currently key challenges for many brick-and-mortar stores. In particular, quick and accurate responses are required when handling inquiries and troubleshooting, which directly impacts the customer experience. However, handling a large number of inquiries requires human intervention, making it difficult to secure and manage a workforce. Furthermore, while properly understanding customer emotions and responding accordingly could contribute to further improving customer satisfaction, automation remains lacking in this area.

[2020] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for receiving an image or electronic document of an invoice; means for converting the received document into character data using optical character recognition technology; means for analyzing the converted character data using generative artificial intelligence and extracting each item of the invoice; means for comparing the extracted invoice issuer information with a database of qualified invoice issuers; means for determining whether the invoice is a qualified invoice based on the comparison result; means for storing the determined invoice as electronic data on a remote platform; means for generating a response to a user inquiry using generative artificial intelligence; and means for adjusting the response based on the emotion data using an emotion recognition engine that analyzes the user's emotions. This makes it possible for brick-and-mortar stores to automatically provide optimal responses based on the analysis of receipt and invoice information and the customer's emotions.

[2021] "Invoice image or electronic document" refers to an image file or a digital document such as a PDF that visually represents the contents of the invoice.

[2022] "Optical character recognition technology" is a technology for analyzing character information contained in an image and converting it into text data.

[2023] "Generative artificial intelligence" is an artificial intelligence technology that generates new data and responses based on large amounts of data, and is used for natural language generation and various data analyses.

[2024] The "Qualified Invoice Issuer Database" is a database that compiles information on businesses that have the authority to issue qualified invoices.

[2025] "Remote infrastructure" refers to cloud environments and remote data storage accessible via the Internet.

[2026] An "emotion recognition engine" is an engine that analyzes and identifies a user's emotional state from data such as text and voice.

[2027] "User emotion" refers to the user's psychological state and emotional expression during an inquiry or conversation.

[2028] "Response" or "response generation" is the process of automatically creating an appropriate reply to a user's inquiry using generative artificial intelligence.

[2029] System Overview

[2030] The system of the present invention includes a series of processes, from reading invoices to determining whether they are eligible, converting them into electronic data, storing them, responding to inquiries, and recognizing the user's emotions to generate the most appropriate response. This system works in cooperation with the server, terminals, and users, and makes full use of a variety of technologies, including OCR technology, generative artificial intelligence, and an emotion recognition engine.

[2031] Server Features

[2032] The server has the following main functions:

[2033] 1. Receiving invoices: Receiving invoice images and electronic documents uploaded by users.

[2034] 2. Optical character recognition processing: The received document is converted into character data using optical character recognition technology (such as pytesseract).

[2035] 3. Data analysis and item extraction: Using generative artificial intelligence (such as Hugging Face's transformers library), each item of the invoice (e.g., invoice number, issue date, amount, issuer information, etc.) is analyzed and extracted from the converted text data.

[2036] 4. Database matching: The extracted invoice issuer information is matched with the database of eligible invoice issuers to determine whether the invoice is eligible.

[2037] 5. Digitalization and storage: The judged invoices are stored as electronic data on a cloud platform.

[2038] 6. Inquiry handling and response generation: Generative artificial intelligence is used to generate appropriate responses to user inquiries.

[2039] 7. Sentiment Analysis: An emotion recognition engine is used to analyze the user's emotional data and tailor the response.

[2040] Device Features

[2041] The terminal provides the following functions to the user:

[2042] 1. Provide an interface: Through the user interface, you can upload invoices, send and receive inquiries, etc.

[2043] 2. Sending and receiving data: Two-way communication with the server to send or receive data.

[2044] User operations

[2045] The user does the following:

[2046] 1. Uploading invoices: Use the terminal to upload an image or electronic document of the invoice to the system.

[2047] 2. Sending a query: Use the terminal to query the system for the required information.

[2048] 3. Check the matching result: Check the matching result sent from the server.

[2049] Hardware and software used

[2050] The hardware used is a customer-facing robot from a brick-and-mortar store (e.g., a typical customer-facing robot), while the software uses OCR technology (pytesseract), generative artificial intelligence (Hugging Face's transformers library and GPT-2 model), and a sentiment analysis module (sentiment_analysis).

[2051] Specific examples

[2052] For example, if a customer asks "Can I get a refund for this item?" in a store, the robot will analyze the customer's facial expressions and tone of voice to determine their emotions, and if the emotion score is low, it will generate a more polite and flexible response. At the same time, if the customer shows the robot a receipt, it will use OCR technology to quickly read the information on the receipt and immediately determine whether a refund is possible and respond accordingly.

[2053] Prompt Sentence Examples

[2054] User Question: "Can I get a refund for this item?"

[2055] Depending on the sentiment analysis results, the generated response may be:

[2056] "Of course, I'm happy to refund this item." (High sentiment score)

[2057] "Sorry, but we need to go through some checks before we can process your refund. Could you please wait a moment?" (if your sentiment score is low)

[2058] This system is expected to improve the efficiency of customer service in physical stores and increase customer satisfaction.

[2059] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2060] Step 1:

[2061] The server receives the invoice image or electronic document uploaded by the user. The input format can be an image file, a PDF file, or other format. The received file is stored in a temporary storage area.

[2062] Step 2:

[2063] The server converts the received invoice into text data using optical character recognition (OCR). It scans the image or PDF file of the invoice received as input and generates text data as output. Specifically, it uses pytesseract to analyze the characters in the image or PDF and convert them into text format.

[2064] Step 3:

[2065] The server uses generative artificial intelligence to analyze and extract each item of the invoice (e.g., invoice number, issue date, amount, issuer information, etc.) from the converted character data. It uses text data generated by OCR as input and generates structured data containing each item of the invoice as output. Specifically, it uses a generative artificial intelligence model using Hugging Face's transformers library.

[2066] Step 4:

[2067] The server compares the extracted invoice issuer information with the database of qualified invoice issuers. It uses the issuer information and database information as input and obtains the matching result as output. This completes the process of determining whether the invoice is qualified.

[2068] Step 5:

[2069] Based on the judgment results, the server stores qualified invoices as electronic data on a cloud platform. Using the matching results and invoice data as input, it generates a message indicating that the data has been saved as output. The electronic data is securely stored in cloud storage and can be accessed by users.

[2070] Step 6:

[2071] When a user makes a query using a terminal, the server receives the query content. At this time, the query content is sent as input and stored on the server side for processing.

[2072] Step 7:

[2073] The server uses generative artificial intelligence to generate appropriate responses to user queries. It uses the user's query as input and obtains the generated response text as output. Specifically, it uses Hugging Face's transformers library to generate the optimal response corresponding to the query.

[2074] Step 8:

[2075] The server uses an emotion recognition engine to analyze the user's emotion data. It uses the query content, the user's tone of voice, and text data as input, and generates an emotion score as output. It uses a sentiment analysis module (such as Sentiment Analysis).

[2076] Step 9:

[2077] The server adjusts the content and tone of the response based on the analyzed emotion data. It uses the emotion score and the initial response text as input and generates an optimized response text as output, which provides an appropriate response that corresponds to the user's emotion.

[2078] Step 10:

[2079] The server sends the generated response to the terminal and provides it to the user, using the optimized response text as input and displaying it on the terminal as output, after which the user can check the generated response.

[2080] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2081] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2082] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2083] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2084] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2085] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2086] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2087] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2088] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2089] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2090] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2091] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2092] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2093] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2094] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2095] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2096] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2097] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2098] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2099] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2100] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2101] The following is further disclosed regarding the above embodiment.

[2102] Below, we propose several draft claims focusing on the distinctive features of the system.

[2103] (Claim 1)

[2104] A means of receiving an image or PDF file of the invoice;

[2105] A means for converting the received file into text data using OCR technology;

[2106] A means for analyzing the converted text data using the generation AI and extracting each item of the invoice;

[2107] means for checking the extracted bill issuer information against a database of qualified bill issuers;

[2108] a means for determining whether the invoice is a qualified invoice based on the matching result;

[2109] A means for storing the judged invoice as electronic data on a cloud platform;

[2110] A means for generating a response to a user's inquiry using a generation AI;

[2111] A system including:

[2112] (Claim 2)

[2113] The system of claim 1, wherein the system automatically accesses and checks the database of qualified billing entities.

[2114] (Claim 3)

[2115] The system according to claim 1, wherein each process of receiving, OCR processing, analysis, matching, judgment, and storage is consistently performed on the cloud.

[2116] (Claim 4)

[2117] 2. The system according to claim 1, wherein access control is performed on the stored billing data based on user authority.

[2118] (Claim 5)

[2119] 2. The system according to claim 1, wherein invoice data is transmitted and stored using encrypted communication using TLS / SSL.

[2120] (Claim 6)

[2121] 10. The system of claim 1, wherein the system automatically performs daily data backups to ensure data safety.

[2122] "Example 1"

[2123] (Claim 1)

[2124] means for receiving a document file of an invoice;

[2125] means for converting the received file into character data using optical character recognition technology;

[2126] A means for analyzing the converted text data using the generation AI and extracting each item of the invoice;

[2127] means for matching the extracted bill issuer information with a database of eligible bill issuers;

[2128] a means for determining whether the invoice is a qualified invoice based on the matching result;

[2129] a means for storing the adjudicated invoice as electronic data on a cloud platform;

[2130] A means for generating a response to a user's inquiry using a generation AI;

[2131] A system including:

[2132] (Claim 2)

[2133] The system of claim 1, wherein the system automatically accesses and checks the database of qualified bill issuers.

[2134] (Claim 3)

[2135] The system according to claim 1, wherein each process of receiving, optical character recognition processing, analysis, matching, judgment, and storage is consistently performed on the cloud.

[2136] "Application Example 1"

[2137] (Claim 1)

[2138] means for receiving an image file of the invoice or label;

[2139] A means for converting the received file into text data using OCR technology;

[2140] A means for analyzing the converted text data using a generation AI and extracting each item;

[2141] means for comparing the extracted issuer information or product information with an eligibility determination database;

[2142] a means for determining whether the issue or product is eligible based on the matching results;

[2143] a means for storing the determined publication or product information as electronic data in a cloud platform;

[2144] A means for generating a response to a user's inquiry using a generation AI;

[2145] A system including:

[2146] (Claim 2)

[2147] The system of claim 1, wherein the system automatically accesses and checks the eligibility determination database.

[2148] (Claim 3)

[2149] The system according to claim 1, wherein each process of receiving, OCR processing, analysis, matching, judgment, and storage is consistently performed on the cloud.

[2150] "Example 2: Combining Emotion Engines"

[2151] (Claim 1)

[2152] means for receiving an image or electronic file of the invoice;

[2153] means for converting the received file into character data using optical character recognition technology;

[2154] A means for analyzing the converted text data using a generative AI model and extracting each item of the invoice;

[2155] means for checking the extracted bill issuer information against a database of qualified bill issuers;

[2156] a means for determining whether the invoice is a qualified invoice based on the matching result;

[2157] A means for storing the judged invoice as electronic data on a cloud platform;

[2158] A means for generating a response to a user inquiry using a generative AI model;

[2159] using a sentiment analysis engine to analyze the user's sentiment and adjust the content and tone of the response;

[2160] A means of storing emotion analysis data in a learning database to improve the response accuracy of the generative AI model;

[2161] A system including:

[2162] (Claim 2)

[2163] The system of claim 1, wherein the system automatically accesses and checks the database of qualified billing entities.

[2164] (Claim 3)

[2165] The system according to claim 1, wherein each of the steps of receiving, optical character recognition processing, analysis, matching, judgment, storage, query response generation, and sentiment analysis is consistently performed on the cloud.

[2166] "Application example 2 when combining emotion engines"

[2167] Claims

[2168] (Claim 1)

[2169] means for receiving an image or electronic document of the invoice;

[2170] means for converting the received document into character data using optical character recognition technology;

[2171] A means for analyzing the converted character data using generative artificial intelligence and extracting each item of the invoice;

[2172] means for checking the extracted bill issuer information against a database of qualified bill issuers;

[2173] a means for determining whether the invoice is a qualified invoice based on the matching result;

[2174] a means for storing the determined invoice as electronic data on a remote base;

[2175] means for generating responses to user queries using generative artificial intelligence;

[2176] means for adjusting a response based on the emotion data using an emotion recognition engine that analyzes the user's emotions;

[2177] A system including:

[2178] (Claim 2)

[2179] 10. The system of claim 1, wherein the system automatically accesses and checks a database of qualified billing entities and takes into account user sentiment analysis results in tailoring responses.

[2180] (Claim 3)

[2181] 10. The system of claim 1, wherein the steps of receiving, optical character recognition processing, analyzing, matching, determining, storing, and sentiment analysis and response generation are performed remotely and consistently. [Explanation of symbols]

[2182] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of receiving an image or PDF file of the invoice; A means for converting the received file into text data using OCR technology; A means for analyzing the converted text data using the generation AI and extracting each item of the invoice; means for checking the extracted bill issuer information against a database of qualified bill issuers; a means for determining whether the invoice is a qualified invoice based on the matching result; A means for storing the judged invoice as electronic data on a cloud platform; A means for generating a response to a user's inquiry using a generation AI; A system including:

2. 10. The system of claim 1, wherein the system automatically accesses and checks the database of qualified billing entities.

3. The system according to claim 1, wherein each process of receiving, OCR processing, analysis, comparison, judgment, and storage is consistently performed on the cloud.

4. 2. The system according to claim 1, wherein access control is performed on the stored billing data based on user authority.

5. 2. The system according to claim 1, wherein the billing data is transmitted and stored using encrypted communication using TLS / SSL.

6. 10. The system of claim 1, wherein daily data backups are automatically performed to ensure data safety.

Images