System

Abstract

A system is provided.SOLUTION: A system comprising: upload means for a user to capture an image of a document and convert the image into data; character recognition means for a server to extract character information from the uploaded image; summary generation means for the server to summarize the extracted character information; calendar registration means for the server to automatically register summary information and related date information in a calendar application; and question and answer means for the user to pose a question to the system and to answer the question from the summary information or original data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's world, parents, especially those of elementary school students, face the burden of organizing and managing the large amount of printed materials distributed by schools (class newsletters, school lunch menus, parent notices, etc.). With the increasing number of dual-income households, the time and effort required to manage these printed materials is causing significant stress for parents. Furthermore, there is an increased risk of missing important information or forgetting school event schedules. A system that can solve this problem and enable parents to manage information efficiently is needed. [Means for solving the problem]

[0005] The system includes an uploading unit that allows users to take images of documents and digitize them; a character recognition unit that allows a server to extract text information from the uploaded images; a summary generation unit that allows the server to summarize the extracted text information; a calendar registration unit that allows the server to automatically register the summary information and related date information in a calendar application; and a question and answer unit that allows users to ask questions to the system and receive answers from the summary information and the original data. This system allows users to summarize, manage, and answer questions about print content simply by taking a photo, significantly reducing the burden of organizing prints. The character recognition unit uses optical character recognition (OCR) and the summary generation unit uses a generative model (e.g., GPT) to achieve highly accurate data processing and efficient information management.

[0006] "User" refers to a person who uses this system, and is generally a parent who manages the school handouts that their child takes home.

[0007] "Documents" mainly refer to printed materials distributed by schools, including class newsletters, school lunch menus, and notices addressed to parents.

[0008] "Image" refers to photographic data of a document taken by a user using a smartphone or camera.

[0009] "Digitization" refers to converting analog information (in this case, the contents of a document) into digital form.

[0010] "Uploading means" refers to the function or method for sending images taken by a user to a server.

[0011] "Server" refers to the computer system that processes and stores data and manages and operates the entire System.

[0012] "Text information" refers to text data written on a document.

[0013] "Character recognition means" refers to techniques and methods for extracting character information from images, and specifically includes optical character recognition (OCR) technology.

[0014] "Summary generation means" refers to the technology or method for generating a summary based on extracted text information, specifically using a generative model (e.g., GPT).

[0015] "Calendar application" refers to software that allows you to register and manage appointments and reminders.

[0016] "Calendar registration means" refers to a function or method for automatically registering summary content and related date information in a calendar application.

[0017] "Question and answer means" refers to a function or method that allows a user to ask a question to the system and receive an answer to that question from summary information or original data.

[0018] "Optical character recognition technology (OCR)" refers to the technology that recognizes characters from an image and converts them into text data.

[0019] A "generative model" refers to a machine learning model that generates new text data based on input data, and specifically includes natural language processing technology. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] This invention is a system for efficiently managing a large amount of printed materials distributed by a school, and is specifically implemented as follows.

[0042] Overall system overview

[0043] The system automates a series of processes: users upload images of prints they have taken, extract text information from the images to generate summaries, register them in a calendar application, and then participate in a Q&A session.

[0044] Program processing

[0045] 1. Taking a photo and uploading it

[0046] The user takes a photo of the printouts their child brings home using their smartphone. After taking the photo, the user uploads the image to the server using a dedicated app for the system. By pressing the upload button, the image data is sent to the server.

[0047] 2. Image Processing and Character Recognition

[0048] The server temporarily stores the received image for processing. The server then uses OCR (optical character recognition) technology to recognize the characters in the image and convert them into text data. This process digitizes the contents of the paper printout.

[0049] 3. Summary of Contents

[0050] The server summarizes the text information extracted by OCR using a generative model (e.g., GPT). This summary generation method produces a short and concise summary of the long printed content.

[0051] 4. Register with your calendar app

[0052] The server analyzes the summarized information and automatically registers relevant dates and event information in a calendar application. For example, if the content says "The sports day will be held on the school playground on July 15th," the date "July 15th" will be identified and registered as an event called "Sports Day" in the calendar.

[0053] 5. Question and Answer System

[0054] The server stores the summary and the original text information in a database. When a user enters a question into the system using a smartphone or PC, the server searches the database for an answer to that question. For example, in response to a question such as "Where will the sports day be held?", the server will return an answer such as "The sports day will be held in the school's athletic field."

[0055] Specific examples

[0056] Example 1: Sports day announcement

[0057] A user takes a photo of a notice printout for a sports day and uploads it to the system. The server extracts the text information "The sports day will be held on July 15th at the school playground" from the image and summarizes it as "Sports day: July 15th" using a summary generation model. After that, an event called "Sports day" is registered in the calendar app for July 15th. When the user asks "What time does the sports day start?", the server replies "The sports day starts at 9:00 AM."

[0058] Example 2: School lunch menu

[0059] A user takes a photo of a school lunch menu and uploads it to the system. The server extracts text information from the image, such as "July 1st: curry rice, July 2nd: fish meunière," and summarizes it as "July menu" using a summary generation model. The calendar is registered with the school lunch menu for each date. When a user asks, "What's for lunch on July 2nd?", the server responds, "The school lunch on July 2nd is fish meunière."

[0060] This system allows users to quickly understand the contents of printed materials, making it easier to manage schedules. It also allows users to quickly get answers to their questions, significantly reducing the burden of organizing printed materials.

[0061] The processing flow will be explained below.

[0062] Step 1:

[0063] The user takes a photo of the print that the child brought home using a smartphone terminal.

[0064] Step 2:

[0065] The images taken by the terminal are imported into the "School Print Summary System" application.

[0066] Step 3:

[0067] The user presses the "Upload" button in the app to send the image data to the server.

[0068] Step 4:

[0069] The server receives the uploaded image data and temporarily stores it.

[0070] Step 5:

[0071] The server invokes the OCR engine to extract text information from the stored image data, specifically recognizing text areas within the image and converting them into text data.

[0072] Step 6:

[0073] The server uses a generative AI model (e.g., GPT) to summarize the extracted text information. Specifically, it extracts key points from long text data and generates a concise summary.

[0074] Step 7:

[0075] The server parses the summarized information to identify relevant date and event information, specifically identifying dates and event names within the text.

[0076] Step 8:

[0077] The server calls the calendar application's API and automatically registers the specified events and dates in the calendar. For example, using the Google Calendar API, it registers "Sports Day" on July 15th.

[0078] Step 9:

[0079] The server stores the summary information and the original text in a database, allowing for later question and answer sessions.

[0080] Step 10:

[0081] A user opens a chat window within the system and types a question, for example, "Where is the sports day being held?"

[0082] Step 11:

[0083] The server receives the user's question, searches for relevant information in the database, and generates the best answer based on the summary information and the original text information.

[0084] Step 12:

[0085] The server sends the generated answer to the user, for example, "The sports day will be held on the school grounds."

[0086] The system operates according to these steps, allowing users to efficiently manage print content and quickly obtain the information they need.

[0087] Example 1

[0088] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0089] Conventional document management systems lacked the means to efficiently manage the large number of printed materials distributed by schools, requiring users to manually check the contents and manage schedules. Furthermore, they were unable to effectively combine question and answer functions, making it difficult to quickly obtain information. This resulted in significant waste of time and effort.

[0090] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0091] In this invention, the server includes a prompt generation means for generating a prompt sentence based on character information acquired by OCR using a generative AI model, a summary generation means for sending the generated prompt sentence to the generative AI model and generating a summary, and a calendar registration means for automatically registering the summary information and related date information in a calendar application, thereby enabling a user to quickly understand the contents of a document, efficiently manage their schedule, and quickly obtain necessary information.

[0092] "User" refers to a person who uses the system to take images of documents and review and manage information.

[0093] "Documents" refers to physical documents such as handouts and notices distributed by the school.

[0094] "Image" refers to the digital photograph obtained when a user photographs a document.

[0095] "Server" refers to the central system that processes images uploaded by users and performs character recognition, summary generation, calendar registration, and question and answering.

[0096] "Uploading means" refers to a function or device for sending images taken by a user to a server.

[0097] "Character recognition means" refers to a function or device that extracts character information from uploaded images.

[0098] "Summary generation means" refers to a function or device that briefly summarizes extracted text information.

[0099] "Calendar registration means" refers to a function or device that automatically registers summarized information and related date information in a calendar application.

[0100] "Question and answering means" refers to a function or device that answers questions posed by users to the system using summary information or original data.

[0101] "OCR" stands for optical character recognition technology, which refers to the technology of extracting text information from images.

[0102] "Prompt generation means" refers to a function or device that generates a prompt sentence based on character information obtained by OCR.

[0103] A "generative AI model" refers to an artificial intelligence model that performs natural language processing and enables summarization and question-and-answering.

[0104] This invention is a system for efficiently managing large volumes of documents distributed by schools. This system automates a series of processes: users take photos of documents, upload them to a server, extract text information using OCR technology, generate summaries using a generative AI model, register the summaries in a calendar application, and then handle questions and answers.

[0105] Hardware and software used

[0106] Terminal

[0107] Users use their smartphones to take pictures of documents and then upload the images to the server using a dedicated application for the system.

[0108] server

[0109] The server implements the following functions:

[0110] 1. Character recognition means

[0111] The server uses OCR (Optical Character Recognition) technology, such as the Tesseract OCR library, to extract text information from the uploaded image.

[0112] 2. Prompt Generation Method

[0113] The server generates a prompt based on the text information obtained by OCR, which is then sent to a generative AI model to generate a summary.

[0114] 3. Summary generation means

[0115] The server summarizes text information using a generative AI model such as GPT-3. The server inputs the prompt sentence generated by the prompt generation means into the model and obtains the summary result.

[0116] 4. Calendar registration method

[0117] The server analyzes the summarized information to find relevant dates and events, and automatically registers them in a calendar application. For example, by using the Google Calendar API, extracted event information can be added to a calendar.

[0118] 5. Q&A tools

[0119] The server stores the summary information and the original text information in a database and provides appropriate answers to questions from users. When a user enters a question on a smartphone or PC, the server searches the database and returns the relevant information.

[0120] Specific examples

[0121] Example 1: Sports day announcement

[0122] A user takes a photo of a sports day announcement printout and uploads it to the system. The server extracts the text information "The sports day will be held on July 15th at the school playground" from the image and summarizes it as "Sports day: July 15th" using a generative AI model. Then, an event called "Sports day" is registered in a calendar app for July 15th. When the user asks, "What time does the sports day start?", the server responds with, "The sports day starts at 9:00 AM."

[0123] Example 2: School lunch menu

[0124] A user takes a photo of a school lunch menu and uploads it to the system. The server extracts text information from the image, such as "July 1st: Curry rice, July 2nd: Fish meunière," and uses a generative AI model to summarize it as "July menu." The calendar is then populated with school lunch menus for each date. When a user asks, "What's for lunch on July 2nd?", the server responds, "The school lunch on July 2nd is fish meunière."

[0125] Prompt Sentence Examples

[0126] An example prompt for a generative AI model to summarize is:

[0127] Example prompt 1:

[0128] "Please summarize the following sentence concisely: 'The sports day will be held on July 15th at the school playground. It will start at 9:00 AM. Participants should assemble in their respective classes.'"

[0129] Example prompt 2:

[0130] "Based on the following information, please generate a summary for calendar registration: 'July 1st: Curry rice, July 2nd: Fish meunière, July 3rd: Hamburg steak'"

[0131] This system allows users to quickly understand the contents of printed materials, making it easier to manage schedules. It also allows users to quickly get answers to their questions, significantly reducing the burden of organizing printed materials.

[0132] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0133] Step 1:

[0134] The user uses a smartphone to take a photo of the document that the child has brought home, opens the dedicated app, selects the image, and presses the upload button, which sends the image data to the server.

[0135] Input: Image data of a document photographed with a smartphone

[0136] Output: Image data sent to the server

[0137] Specific behavior:

[0138] The user launches the camera app on their smartphone and takes a photo of the document.

[0139] The user opens the system's dedicated app and selects the photos they have taken.

[0140] The user presses the upload button in the app to send the image data to the server.

[0141] Step 2:

[0142] When the server receives the uploaded image data, it temporarily stores it, then uses OCR (optical character recognition) technology to extract character information from the image and convert it into text data.

[0143] Input: Image data uploaded to the server

[0144] Output: Extracted character information (text data)

[0145] Specific behavior:

[0146] The server temporarily stores the image data in a defined directory.

[0147] The server runs an OCR engine (e.g., Tesseract) to extract text information from the image.

[0148] The extracted character information is saved as a text file and passed on to the next process.

[0149] Step 3:

[0150] The server generates a prompt sentence based on the character information acquired by OCR and the generative AI model, and sends this prompt sentence to the generative AI model to obtain a summary.

[0151] Input: Character information obtained by OCR

[0152] Output: Generated prompt statement and summary information

[0153] Specific behavior:

[0154] The server generates a prompt based on the character information obtained by OCR.

[0155] The server sends prompt sentences to the generative AI model and obtains summaries.

[0156] The generated summary information is passed to the next process.

[0157] Step 4:

[0158] The server analyzes the summarized information to find relevant dates and event information and automatically registers them in the calendar application.

[0159] Input: Summarized information (text data)

[0160] Output: Event information registered in the calendar

[0161] Specific behavior:

[0162] The server parses the dates and event names contained in the summary.

[0163] The server uses a calendar API (e.g., Google Calendar API) to add the event information to the calendar.

[0164] Step 5:

[0165] The server stores summary information and the original text information in a database. When a user enters a question on a smartphone or computer, the server searches the database and provides the appropriate answer.

[0166] Input: Question from user (text data)

[0167] Output: Answer to the question (text data)

[0168] Specific behavior:

[0169] The server stores the OCR and summary information in a database.

[0170] Users enter their questions on their smartphones or computers.

[0171] The server searches a database and returns an answer based on the question entered.

[0172] (Application example 1)

[0173] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0174] In brick-and-mortar stores, it is difficult to efficiently manage paper media such as shift schedules and promotion information used by store staff in their daily work. This results in problems such as time-consuming sharing and updating of information, which reduces work efficiency. There is also a lack of means to quickly search for necessary information and obtain answers. There is a need for a system that can solve these problems.

[0175] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0176] In this invention, the server includes an uploading means for a user to photograph information on paper media as an image and convert the image into data, a character recognition means for the server to extract text information from the uploaded image, a summary generation means for the server to summarize the extracted text information, a calendar registration means for the server to automatically register the summary information and related date information in a calendar application, a question and answer means for a user to ask the system questions and the system answers the questions using the summary information or original data, and a store data linking means for linking the extracted text information and summary to a store management system. This enables store staff to efficiently digitize information on paper media and quickly search and manage it.

[0177] "User" refers to a person who uses the system to digitize and manage paper-based information.

[0178] "Paper media" refers to physical documents or books on which text, charts, etc. are printed.

[0179] The "uploading means" refers to a function that allows a user to take an image of information on a paper medium and send the image data to a server.

[0180] "Character recognition means" refers to the function by which the server extracts character information from uploaded images, and typically uses optical character recognition technology.

[0181] "Summary generation means" refers to the function by which the server succinctly summarizes the extracted text information, and typically uses a generative AI model.

[0182] The "calendar registration means" refers to a function that automatically registers summary information and related date information in a calendar application.

[0183] "Question and answer means" refers to a function that allows users to ask questions to the system and receive answers from summary information or original data.

[0184] "Store data linking means" refers to the function of associating extracted text information and summaries with a store management system.

[0185] "Server" refers to a computer device that processes, stores, and performs corresponding functions for the entire system.

[0186] "Physical store" refers to a store that exists in a physical location and is managed and operated by store staff.

[0187] The present invention is a system for efficiently managing paper-based information in a physical store, and is specifically implemented as follows.

[0188] System Configuration

[0189] This system automates the process by which users digitize paper media in their stores (e.g., shift schedules, promotion information) and register that information in a store management system. Specifically, it consists of devices such as smartphones and tablets, a server, and various software.

[0190] Hardware and Software

[0191] Hardware:

[0192] Smartphone or tablet

[0193] server

[0194] software:

[0195] Image processing library (OpenCV)

[0196] Optical Character Recognition Library (Tesseract-OCR)

[0197] Generative AI model (HuggingFace Transformers T5 model)

[0198] Calendar event generation library (icalendar)

[0199] Operating procedure

[0200] 1. The user photographs a paper medium: The user uses a smartphone or tablet to take a picture of a paper medium such as a shift schedule or promotional information.

[0201] 2. Uploading image data: The captured image data is uploaded to the server via a dedicated application.

[0202] 3. Character recognition by the server: The server preprocesses the received image using an image processing library (OpenCV) and extracts character information using an optical character recognition library (Tesseract-OCR).

[0203] 4. Summary generation: The extracted text information is summarized using a generative AI model (e.g., the T5 model from HuggingFace Transformers).

[0204] 5. Calendar Registration: The summary information and associated date information are automatically registered in the calendar application using the calendar event generation library (icalendar).

[0205] 6. Linkage to store management system: The extracted text information and summaries are linked to the store management system and reflected in staff work shifts and promotion schedules.

[0206] 7. Question and Answer System: Users can input questions into the system and the server will provide answers to the questions from summary information and original data.

[0207] Specific examples

[0208] Example 1: Managing shift schedules

[0209] A user takes a photo of a shift schedule and uploads the image to the server. The server extracts text information from the image, such as "July 1st: Person A is on duty, July 2nd: Person B is on duty," and summarizes it as the "July shift schedule" using a generative AI model. Shift information for each date is registered on the calendar. When a user asks, "Who is on the July 2nd shift?" the server responds, "Person B is on the July 2nd shift."

[0210] Example prompt sentence:

[0211] What's the next promotion?

[0212] Example 2: Managing promotion information

[0213] A user takes a photo of a printed copy of promotional information and uploads the image to a server. The server extracts text information from the image, such as "10% off sale on January 10th," and summarizes it using a generative AI model. An event called "Promotion: 10% off sale (January 10th)" is then registered in a calendar app. When the user asks, "What's the next promotion?", the server responds, "The next promotion is a 10% off sale on January 10th."

[0214] This system allows staff at physical stores to efficiently digitize paper-based information, enabling smooth business management.

[0215] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0216] Step 1:

[0217] Users take a photo of paper-based information (such as shift schedules or promotional information) using a smartphone or tablet. This generates image data, which is then uploaded to a server via a dedicated application. The input is the captured image data, and the output is the uploaded image data.

[0218] Step 2:

[0219] The server receives the uploaded image data and temporarily stores it. This image data is prepared for the next processing. The input is the uploaded image data, and the output is the stored image data.

[0220] Step 3:

[0221] The server preprocesses the received image data using an image processing library (OpenCV). Specifically, it converts the image to grayscale and performs noise reduction. The input is the stored image data, and the output is the preprocessed image data.

[0222] Step 4:

[0223] The server extracts character information from the preprocessed image using an optical character recognition library (Tesseract-OCR). The input is the preprocessed image data, and the output is text data as character information.

[0224] Step 5:

[0225] The server uses a generative AI model (e.g., the T5 model from HuggingFace Transformers) to summarize the extracted text information. The input is text data as text information, and the output is summarized information.

[0226] Step 6:

[0227] The server uses a calendar event generation library (icalendar) to register the summary information and associated date information to the calendar application. The input is the summary information and date information, and the output is the event registered in the calendar.

[0228] Step 7:

[0229] The server associates the extracted text and summary information with the store management system, which updates shift information and promotion schedules. The input is text and summary information, and the output is updated store management data.

[0230] Step 8:

[0231] The user inputs a question to the system. The server searches for an appropriate answer from the summary information and the original data based on the question and provides it to the user. The input is the user's question, and the output is the answer to the question.

[0232] This allows users to efficiently digitize paper-based information, making it easier to manage information within the store.

[0233] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0234] This invention is a system for efficiently managing a large amount of printed materials distributed by schools, and further has a function for recognizing the user's emotions and adjusting the response accordingly. Specific embodiments of the invention are described below.

[0235] Overall system overview

[0236] This system automates a series of processes: uploading images of prints taken by users, extracting text information from the images to generate summaries, registering them in a calendar application, and answering questions. It also incorporates an emotion engine that recognizes emotions from questions and messages entered by users and adjusts responses accordingly.

[0237] Program processing

[0238] 1. Taking a photo and uploading it

[0239] The user takes a photo of the printouts their child brings home using their smartphone. After taking the photo, the user uploads the image to the server using a dedicated app for the system. By pressing the upload button, the image data is sent to the server.

[0240] 2. Image Processing and Character Recognition

[0241] The server temporarily stores the received image for processing. The server then uses OCR (optical character recognition) technology to recognize the characters in the image and convert them into text data. This process digitizes the contents of the paper printout.

[0242] 3. Summary of Contents

[0243] The server summarizes the text information extracted by OCR using a generative model (e.g., GPT). This summary generation method produces a short and concise summary of the long printed content.

[0244] 4. Register with your calendar app

[0245] The server analyzes the summarized information and identifies relevant dates and events. For example, if the content says "The sports day will be held on the school grounds on July 15th," the date "July 15th" is identified and an event called "Sports Day" is registered on the calendar.

[0246] 5. Question and Answer System

[0247] The server stores the summary and the original text information in a database. When a user enters a question into the system using a smartphone or PC, the server searches the database for an answer to that question. For example, in response to a question such as "Where will the sports day be held?", the server will return an answer such as "The sports day will be held in the school's athletic field."

[0248] 6. Leveraging Emotional Engines

[0249] The server analyzes the questions and messages entered by the user and uses an emotion engine to recognize the user's emotional state. For example, if the user enters "It's too late! Tell me quickly!", the emotion engine will determine that the user is irritated.

[0250] If the emotion engine recognizes the user's emotion, the server will adjust the content and tone of the response, for example, responding to an annoyed user with a polite and prompt "I'm sorry, let me check again."

[0251] Furthermore, a dashboard can be provided to visually display the emotional state recognized by the emotion engine, allowing users to monitor changes in their emotions in real time and respond appropriately.

[0252] Specific examples

[0253] Example 1: Sports day announcement

[0254] A user takes a photo of a sports day announcement printout and uploads it to the system. The server extracts the text information "The sports day will be held on July 15th at the school playground" from the image and summarizes it as "Sports day: July 15th" using a summary generation model. An event called "Sports day" is then registered in the calendar app for July 15th. When the user asks, "What time does the sports day start?", the server replies, "The sports day starts at 9:00 AM." If the user impatiently asks, "Why is it so late?", the server responds politely with, "Sorry, I'll check right away."

[0255] Example 2: School lunch menu

[0256] A user takes a photo of a school lunch menu and uploads it to the system. The server extracts text information from the image, such as "July 1st: Curry rice, July 2nd: Fish meunière," and summarizes it as "July menu" using a summary generation model. The calendar registers the school lunch menu for each date. If a user asks, "What's for lunch on July 2nd?", the server responds, "The school lunch on July 2nd is fish meunière." If the user complains that the server is slow to respond, the server responds, "Sorry for the wait, we'll get back to you right away."

[0257] In this way, by utilizing the emotion engine, this system can efficiently summarize, manage, and answer questions about information while responding appropriately to the user's emotions.

[0258] The processing flow will be explained below.

[0259] Step 1:

[0260] The user takes a photo of the print that the child brought home using a smartphone terminal.

[0261] Step 2:

[0262] The images taken by the terminal are imported into the "School Print Summary System" application.

[0263] Step 3:

[0264] The user presses the "Upload" button in the app to send the image data to the server.

[0265] Step 4:

[0266] The server receives the uploaded image data and temporarily stores it.

[0267] Step 5:

[0268] The server invokes the OCR engine to extract text information from the stored image data, specifically recognizing text areas within the image and converting them into text data.

[0269] Step 6:

[0270] The server uses a generative AI model (e.g., GPT) to summarize the extracted text information. Specifically, it extracts key points from long text data and generates a concise summary.

[0271] Step 7:

[0272] The server parses the summarized information to identify relevant date and event information, specifically identifying dates and event names within the text.

[0273] Step 8:

[0274] The server calls the calendar application's API and automatically registers the specified events and dates in the calendar. For example, using the Google Calendar API, it registers "Sports Day" on July 15th.

[0275] Step 9:

[0276] The server stores the summary information and the original text in a database, allowing for later question and answer sessions.

[0277] Step 10:

[0278] A user opens a chat window within the system and types a question, for example, "Where is the sports day being held?"

[0279] Step 11:

[0280] The server receives the user's question, searches for relevant information in the database, and generates the best answer based on the summary information and the original text information.

[0281] Step 12:

[0282] The server sends the generated answer to the user, for example, "The sports day will be held on the school grounds."

[0283] Step 13:

[0284] The server analyzes the user's questions and messages and uses an emotion engine to recognize the user's emotional state. For example, if the user types "You're late! Tell me quickly!", the server detects the emotion of irritation.

[0285] Step 14:

[0286] The server adjusts the content and tone of the response depending on the emotion recognized by the emotion engine. If it detects irritation, it will respond politely and promptly, saying something like, "I'm sorry, we'll look into it right away."

[0287] Step 15:

[0288] The server reflects the user's emotions recognized by the emotion engine on a visual dashboard, which is used by system administrators to monitor the user's emotional state in real time.

[0289] The system operates according to these steps, allowing users to efficiently manage print content and quickly obtain the information they need. It also provides a better user experience by responding according to the user's emotions.

[0290] Example 2

[0291] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0292] Conventional print management systems have difficulty efficiently digitizing large amounts of paper media and managing summaries and schedules. Responses to user questions are also limited to standardized content, and there are also issues with not being able to recognize user emotions and respond flexibly.

[0293] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0294] In this invention, the server includes uploading means for a user to take an image of a recording medium and digitize the image, character recognition means for the server to extract character information from the uploaded image, summary generation means for the server to summarize the extracted character information, calendar registration means for the server to automatically register the summary information and related date information in a time management application, question and answering means for the user to ask a question to the system and to answer the question from the summary information or original data, emotion recognition means for the server to analyze the question or message entered by the user and recognize the user's emotional state using an emotion engine, and tone adjustment means for adjusting the content and tone of the response based on the recognition result by the emotion recognition means. This enables efficient digitization, summarization, automatic schedule registration, and flexible response based on emotions.

[0295] The "uploading means" is a means by which a user takes an image of a recording medium and transmits the image as data to a server.

[0296] "Character Recognition Means" refers to the techniques or tools used by the server to extract text information from uploaded images.

[0297] "Summary generation" refers to a generative model or other technology that allows the server to summarize extracted textual information in a concise, short format.

[0298] The "calendar registration means" is a means by which the server automatically registers summary information and related date information in the time management application.

[0299] The "question and answer means" is a means by which a user can ask a question to the system and have the system provide an answer based on the summary information or the original data in response to the question.

[0300] "Emotion recognition means" refers to an engine or technology that allows the server to analyze questions and messages entered by the user and recognize the user's emotional state.

[0301] The "tone adjustment means" is a means by which the server responds to the user by adjusting the content and tone of the response based on the recognition result by the emotion recognition means.

[0302] The present invention is a system for efficiently managing a large number of printed materials distributed by schools, recognizing the emotions of users, and adjusting responses accordingly. Specific embodiments of this system are described below.

[0303] Hardware / Software used

[0304] 1. Smartphone device:

[0305] This is a device used by users to take photos of prints and upload the images to a server using a dedicated app.

[0306] 2. Server:

[0307] A computer system that performs a series of processes such as image processing, character recognition, summary generation, calendar entry, and emotion recognition. It also includes a database.

[0308] 3. OCR (Optical Character Recognition) technology:

[0309] This is a technology for extracting text information from images. Specifically, Tesseract OCR is used.

[0310] 4. Generative AI Models:

[0311] This is a generative model for summarizing text information. For example, GPT-4 is used.

[0312] 5. Calendar application:

[0313] It is a time management tool that registers summary information and date information. It uses APIs such as Google Calendar.

[0314] 6. Emotion Recognition Engine:

[0315] This is a technology for analyzing user emotions, and uses sentiment analysis APIs, etc.

[0316] Specific examples

[0317] Example 1: Sports day announcement

[0318] A user takes a photo of a sports day announcement printout and uploads it to the server using a dedicated app. The server performs OCR processing on the received image and extracts the text information, such as "The sports day will be held on July 15th at the school's athletic field." The server then uses a generative AI model to summarize the information as "Sports day: July 15th." The summary information is then registered on the calendar as an event called "Sports day" using the Google Calendar API.

[0319] When a user asks, "What time does the sports day start?", the server searches the database for relevant information and replies, "The sports day starts at 9 a.m." If the user asks impatiently, "Why is it so late?", the emotion recognition engine will recognize the impatience and the server will respond flexibly, saying, "I'm sorry, I'll check right away."

[0320] Example 2: School lunch menu

[0321] The user takes a photo of the school lunch menu and uploads it to the server using a dedicated app. The server performs OCR on the image and extracts the text information "July 1st: Curry rice, July 2nd: Fish meunière." A generative AI model is used to summarize it as "July menu." The school lunch menu for each date is then registered on the calendar.

[0322] When a user asks, "What's for lunch on July 2nd?", the server responds, "The school lunch on July 2nd is fish meunière." If the user expresses dissatisfaction by saying, "You're taking so long to respond!", the emotion recognition engine recognizes the dissatisfaction, and the server responds, "Sorry to keep you waiting, we'll get back to you right away."

[0323] Prompt Sentence Examples

[0324] Example prompt 1:

[0325] "This is a printout announcing the sports day. Please summarize the following information and add it to your calendar. Content: 'The sports day will be held on July 15th at the school playground.'"

[0326] Example prompt 2:

[0327] "I have the school lunch menu in front of me. Please register the following school lunch menu items on your calendar. Contents: July 1st: Curry rice, July 2nd: Fish meunière."

[0328] As described above, this system efficiently digitizes, summarizes, and manages schedules based on user-taken print images. It also recognizes user emotions and responds flexibly accordingly. In this way, advanced OCR technology, generative AI models, and emotion recognition engines work together to improve the effectiveness of the entire system.

[0329] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0330] Step 1:

[0331] Taking photos and uploading

[0332] A user uses a smartphone device to take a photo of a printout that a child has brought home, for example, a printout listing a school event schedule.

[0333] Input: Printed image on paper

[0334] Output: Digital image files (JPEG, PNG, etc.)

[0335] Specific operation: The user launches the camera app on their smartphone, adjusts the screen so that the entire print fits on the screen, and takes a picture. After taking the picture, the image file is saved on the device.

[0336] Step 2:

[0337] Upload

[0338] The user launches the system's dedicated app and uploads the captured image to the server. Specifically, the user taps the "upload" button in the app, and the image is sent to the server.

[0339] Input: Digital image file

[0340] Output: Temporary file on the server

[0341] Specific operation: When the user taps the "Upload" button in the app, the selected image is sent to the server, where it is temporarily stored.

[0342] Step 3:

[0343] Image processing and character recognition

[0344] The server processes the uploaded image using OCR (Optical Character Recognition) technology to extract the text information from the image. Specifically, it uses tools such as Tesseract OCR to generate text data from the image.

[0345] Input: Temporary file on the server (digital image)

[0346] Output: Extracted character information (text data)

[0347] Specific operation: The server reads the saved image file and applies OCR technology to extract text information. The extracted text data is stored on the server.

[0348] Step 4:

[0349] Summary of contents

[0350] The server summarizes the text information extracted by OCR using a generative model (e.g., GPT-4). It provides a prompt such as "Please summarize the following content:"

[0351] Input: Extracted text information

[0352] Output: Summarized text information

[0353] Specific operation: The server inputs the extracted text information into the generative AI model and processes it along with the prompt "Please summarize the following content:" The generated summary is stored on the server.

[0354] Step 5:

[0355] Registering with a calendar app

[0356] The server identifies events and date information from the summarized information and automatically registers them in the calendar using an API such as Google Calendar.

[0357] Input: Summarized text information

[0358] Output: Event information registered in the calendar app

[0359] Specific operation: The server extracts the date, time, and event name from the summary text and registers the identified information in the calendar via the Google Calendar API. For example, "Sports Day: July 15th" is added to the calendar.

[0360] Step 6:

[0361] Q&A System

[0362] Users enter questions into the system using their smartphones or computers, and the server searches a database for answers to those questions and provides them.

[0363] Input: User question (text)

[0364] Output: Answer from the server (text)

[0365] Specific operation: When a user enters a question text through an app or web interface, the server searches the database for relevant information, generates an appropriate answer, and returns it to the user. For example, in response to the question, "What time does the sports day start?", the server responds with, "The sports day starts at 9:00 AM."

[0366] Step 7:

[0367] Utilizing the Emotion Engine

[0368] The server analyzes the questions and messages entered by the user, uses an emotion engine to recognize the user's emotional state, and adjusts the content and tone of the response based on the emotion recognition results.

[0369] Input: User question or message (text)

[0370] Output: Emotion recognition results and tailored responses (text)

[0371] What it does: The server extracts emotions from the user's text through an emotion analysis API, and if it detects irritation or frustration, it responds with a tailored response such as "Sorry, we'll take a look at it right away."

[0372] Emotional states are displayed in real time on a dashboard.

[0373] (Application example 2)

[0374] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0375] Conventional document management systems have difficulty efficiently digitizing and managing large volumes of documents, making the management of factory instructions and manuals particularly cumbersome. Furthermore, they do not respond to the emotions of operators, which can lead to problems such as reduced production efficiency and increased likelihood of work errors. The objective of this invention is to solve these problems by providing a system that responds appropriately to emotions while improving operator efficiency.

[0376] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: uploading means for a user to photograph an image of a document and digitize the image; character recognition means for the server to extract text information from the uploaded image; summary generation means for the server to summarize the extracted text information; calendar registration means for the server to automatically register the summary information and related date information in a calendar application; question and answer means for a user to ask the system questions and the system to answer the questions using the summary information or the original data; emotion analysis means for recognizing the user's emotional state and adjusting the response; means for photographing an image of a document using a photographing device attached to the item processing device; and transmission means for transmitting the photographed image to the processing device. This not only improves the efficiency of document management in the factory but also enables appropriate responses according to the operator's emotions.

[0377] The "uploading means" is a means for converting images taken by a user into data and transmitting the data to a server.

[0378] "Character recognition means" refers to a technology that extracts character information from uploaded images, and refers to optical character recognition technology.

[0379] The "summary generation means" is a technique for concisely summarizing extracted text information using a generative model.

[0380] The "calendar registration means" is a technique for automatically registering summary information and related date information in a calendar application.

[0381] "Question and answering means" is a technology that provides answers to questions posed by users to the system from summary information or original data.

[0382] "Emotion analysis means" is a technology that recognizes the emotional state of a user from questions or messages entered by the user and adjusts the response accordingly.

[0383] "Image capture device" refers to a device attached to an article processing device that captures an image of a document.

[0384] The "transmission means" is a means for transmitting the captured image to the processing device.

[0385] "Image processing means" refers to a series of technologies that analyze uploaded images and convert them into data.

[0386] The "information management system" is a series of systems that upload users' images, extract information from those images, summarize it, register it on a calendar, answer users' questions, and analyze users' emotions to adjust the response.

[0387] The following describes an embodiment of the present invention: This system uses smart glasses to capture images of documents and instructions used in a factory, digitize them, and manage them.

[0388] System Overview

[0389] An operator in a factory wears smart glasses and takes a photo of an instruction manual or instruction sheet. The captured image is processed by a dedicated application in the smart glasses and then uploaded to a server. The server extracts text information from the image, summarizes it, and registers it in a task management system. It also responds to the operator's questions and adjusts its response according to the operator's emotional state.

[0390] Hardware Configuration

[0391] Smart glasses: Worn by operators in the factory, they take photos of instructions and manuals.

[0392] Camera: Built into the smart glasses and equipped with a high-resolution camera.

[0393] Cloud server: Performs image processing, character recognition, summary generation, calendar registration, and sentiment analysis.

[0394] Task Management System: Functions as a calendar application, managing summary information and related date information.

[0395] Software Configuration

[0396] OCR software: Uses software such as Google Cloud Vision to extract text information from uploaded images.

[0397] Generative AI model: Summarize text information using OpenAI's GPT model.

[0398] Sentiment analysis engine: Using IBM Watson NLU and other technologies, it recognizes the emotional state of the user from their questions and messages and adjusts the response accordingly.

[0399] Dedicated application: Installed on smart glasses, it provides functions for taking photos, uploading images, and answering questions.

[0400] Processing flow

[0401] The server includes an uploading means for a user to take an image of a document and convert the image into data, a character recognition means for the server to extract character information from the uploaded image, a summary generation means for the server to summarize the extracted character information, a calendar registration means for the server to automatically register the summary information and related date information in a calendar application, a question and answer means for a user to ask a question to the system and the question is answered from the summary information or original data, an emotion analysis means for recognizing the emotional state of the user and adjusting the response, a means for taking an image of a document using a camera attached to the item processing device, and a transmission means for transmitting the taken image to the processing device.

[0402] Specific examples

[0403] For example, when an operator takes a photo of an instruction sheet using smart glasses, the image is first saved in a dedicated app on the smart glasses and then uploaded to a server. The server uses Google Cloud Vision for OCR processing to extract text information. The text is then summarized using OpenAI's GPT model, and the results are registered in the factory's task management system. When an operator asks, "What's the next task?", the Q&A system provides an answer based on the summary information and the original data. If the operator asks impatiently, "Why is it so slow?", IBM Watson NLU performs sentiment analysis and provides a polite response such as, "Sorry, we'll get back to you right away."

[0404] Prompt Sentence Examples

[0405] Summarize the following text:

[0406] The sports day will be held on July 15th at the school playground.

[0407] summary:

[0408] In this way, the invention makes it possible to improve the efficiency of document management within a factory and also to respond appropriately to the emotions of operators.

[0409] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0410] Process step description

[0411] (Step 1: User takes a photo of the document)

[0412] A user puts on the smart glasses and takes a photo of an instruction manual or instruction sheet. The input is the photographed paper document, and the output is a digital image file. Specifically, the glasses use a high-resolution camera to capture a clear image.

[0413] (Step 2: Save the image with the smart glasses)

[0414] The digital image file is stored in a dedicated application in the smart glasses. The input is the captured digital image, and the output is the image file stored in the smart glasses' storage. The specific operation is to perform the image file storage process.

[0415] (Step 3: Upload the image to the server)

[0416] The saved image files are uploaded to a server by a dedicated application in the smart glasses. The input is the image file saved in the smart glasses' storage, and the output is the image data sent to the server. Specifically, the data is sent via an internet connection.

[0417] (Step 4: Image processing and OCR on the server)

[0418] The server uses OCR software such as Google Cloud Vision to extract text information from uploaded images. The input is the image data uploaded to the server, and the output is text data with the text information. Specifically, OCR processing is performed to recognize the characters in the image.

[0419] (Step 5: Summary generation)

[0420] The server uses the character information extracted by OCR to generate a summary using OpenAI's GPT model. The input is text data extracted by OCR, and the output is a summarized text. The specific operation involves natural language processing using a generative AI model.

[0421] (Step 6: Register with your calendar application)

[0422] The server analyzes the summarized text information and related date information and automatically registers it in the factory's task management system. The input is the summarized text data, and the output is event information registered in a calendar application. Specific operations include text analysis and database registration.

[0423] (Step 7: Question and Answer System)

[0424] A user inputs a question into smart glasses or other devices, and the question is sent to a server. The server uses a question-and-answer system to generate an answer from summary information and the original data. The input is the user's question, and the output is the generated answer. Specifically, the system uses natural language processing technology to analyze the question and generate the answer.

[0425] (Step 8: Sentiment analysis and response adjustment)

[0426] The server analyzes the questions and messages entered by the user and uses an emotion analysis engine such as IBM Watson NLU to recognize the user's emotional state and adjust the response. The input is the user's question or message, and the output is a response that adjusts the emotional state and response. The specific operation involves emotion analysis and adjusting the response based on that.

[0427] This system not only improves the efficiency of document management within the factory, but also enables appropriate responses based on the emotions of operators.

[0428] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0429] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0430] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0431] [Second embodiment]

[0432] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0433] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0434] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0435] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0436] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0437] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0438] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0439] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0440] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0441] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0442] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0443] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0444] This invention is a system for efficiently managing a large amount of printed materials distributed by a school, and is specifically implemented as follows.

[0445] Overall system overview

[0446] The system automates a series of processes: users upload images of prints they have taken, extract text information from the images to generate summaries, register them in a calendar application, and then participate in a Q&A session.

[0447] Program processing

[0448] 1. Taking a photo and uploading it

[0449] The user takes a photo of the printouts their child brings home using their smartphone. After taking the photo, the user uploads the image to the server using a dedicated app for the system. By pressing the upload button, the image data is sent to the server.

[0450] 2. Image Processing and Character Recognition

[0451] The server temporarily stores the received image for processing. The server then uses OCR (optical character recognition) technology to recognize the characters in the image and convert them into text data. This process digitizes the contents of the paper printout.

[0452] 3. Summary of Contents

[0453] The server summarizes the text information extracted by OCR using a generative model (e.g., GPT). This summary generation method produces a short and concise summary of the long printed content.

[0454] 4. Register with your calendar app

[0455] The server analyzes the summarized information and automatically registers relevant dates and event information in a calendar application. For example, if the content says "The sports day will be held on the school playground on July 15th," the date "July 15th" will be identified and registered as an event called "Sports Day" in the calendar.

[0456] 5. Question and Answer System

[0457] The server stores the summary and the original text information in a database. When a user enters a question into the system using a smartphone or PC, the server searches the database for an answer to that question. For example, in response to a question such as "Where will the sports day be held?", the server will return an answer such as "The sports day will be held in the school's athletic field."

[0458] Specific examples

[0459] Example 1: Sports day announcement

[0460] A user takes a photo of a notice printout for a sports day and uploads it to the system. The server extracts the text information "The sports day will be held on July 15th at the school playground" from the image and summarizes it as "Sports day: July 15th" using a summary generation model. After that, an event called "Sports day" is registered in the calendar app for July 15th. When the user asks "What time does the sports day start?", the server replies "The sports day starts at 9:00 AM."

[0461] Example 2: School lunch menu

[0462] A user takes a photo of a school lunch menu and uploads it to the system. The server extracts text information from the image, such as "July 1st: curry rice, July 2nd: fish meunière," and summarizes it as "July menu" using a summary generation model. The calendar is registered with the school lunch menu for each date. When a user asks, "What's for lunch on July 2nd?", the server responds, "The school lunch on July 2nd is fish meunière."

[0463] This system allows users to quickly understand the contents of printed materials, making it easier to manage schedules. It also allows users to quickly get answers to their questions, significantly reducing the burden of organizing printed materials.

[0464] The processing flow will be explained below.

[0465] Step 1:

[0466] The user takes a photo of the print that the child brought home using a smartphone terminal.

[0467] Step 2:

[0468] The images taken by the terminal are imported into the "School Print Summary System" application.

[0469] Step 3:

[0470] The user presses the "Upload" button in the app to send the image data to the server.

[0471] Step 4:

[0472] The server receives the uploaded image data and temporarily stores it.

[0473] Step 5:

[0474] The server invokes the OCR engine to extract text information from the stored image data, specifically recognizing text areas within the image and converting them into text data.

[0475] Step 6:

[0476] The server uses a generative AI model (e.g., GPT) to summarize the extracted text information. Specifically, it extracts key points from long text data and generates a concise summary.

[0477] Step 7:

[0478] The server parses the summarized information to identify relevant date and event information, specifically identifying dates and event names within the text.

[0479] Step 8:

[0480] The server calls the calendar application's API and automatically registers the specified events and dates in the calendar. For example, using the Google Calendar API, it registers "Sports Day" on July 15th.

[0481] Step 9:

[0482] The server stores the summary information and the original text in a database, allowing for later question and answer sessions.

[0483] Step 10:

[0484] A user opens a chat window within the system and types a question, for example, "Where is the sports day being held?"

[0485] Step 11:

[0486] The server receives the user's question, searches for relevant information in the database, and generates the best answer based on the summary information and the original text information.

[0487] Step 12:

[0488] The server sends the generated answer to the user, for example, "The sports day will be held on the school grounds."

[0489] The system operates according to these steps, allowing users to efficiently manage print content and quickly obtain the information they need.

[0490] Example 1

[0491] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0492] Conventional document management systems lacked the means to efficiently manage the large number of printed materials distributed by schools, requiring users to manually check the contents and manage schedules. Furthermore, they were unable to effectively combine question and answer functions, making it difficult to quickly obtain information. This resulted in significant waste of time and effort.

[0493] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0494] In this invention, the server includes a prompt generation means for generating a prompt sentence based on character information acquired by OCR using a generative AI model, a summary generation means for sending the generated prompt sentence to the generative AI model and generating a summary, and a calendar registration means for automatically registering the summary information and related date information in a calendar application, thereby enabling a user to quickly understand the contents of a document, efficiently manage their schedule, and quickly obtain necessary information.

[0495] "User" refers to a person who uses the system to take images of documents and review and manage information.

[0496] "Documents" refers to physical documents such as handouts and notices distributed by the school.

[0497] "Image" refers to the digital photograph obtained when a user photographs a document.

[0498] "Server" refers to the central system that processes images uploaded by users and performs character recognition, summary generation, calendar registration, and question and answering.

[0499] "Uploading means" refers to a function or device for sending images taken by a user to a server.

[0500] "Character recognition means" refers to a function or device that extracts character information from uploaded images.

[0501] "Summary generation means" refers to a function or device that briefly summarizes extracted text information.

[0502] "Calendar registration means" refers to a function or device that automatically registers summarized information and related date information in a calendar application.

[0503] "Question and answering means" refers to a function or device that answers questions posed by users to the system using summary information or original data.

[0504] "OCR" stands for optical character recognition technology, which refers to the technology of extracting text information from images.

[0505] "Prompt generation means" refers to a function or device that generates a prompt sentence based on character information obtained by OCR.

[0506] A "generative AI model" refers to an artificial intelligence model that performs natural language processing and enables summarization and question-and-answering.

[0507] This invention is a system for efficiently managing large volumes of documents distributed by schools. This system automates a series of processes: users take photos of documents, upload them to a server, extract text information using OCR technology, generate summaries using a generative AI model, register the summaries in a calendar application, and then handle questions and answers.

[0508] Hardware and software used

[0509] Terminal

[0510] Users use their smartphones to take pictures of documents and then upload the images to the server using a dedicated application for the system.

[0511] server

[0512] The server implements the following functions:

[0513] 1. Character recognition means

[0514] The server uses OCR (Optical Character Recognition) technology, such as the Tesseract OCR library, to extract text information from the uploaded image.

[0515] 2. Prompt Generation Method

[0516] The server generates a prompt based on the text information obtained by OCR, which is then sent to a generative AI model to generate a summary.

[0517] 3. Summary generation means

[0518] The server summarizes text information using a generative AI model such as GPT-3. The server inputs the prompt sentence generated by the prompt generation means into the model and obtains the summary result.

[0519] 4. Calendar registration method

[0520] The server analyzes the summarized information to find relevant dates and events, and automatically registers them in a calendar application. For example, by using the Google Calendar API, extracted event information can be added to a calendar.

[0521] 5. Q&A tools

[0522] The server stores the summary information and the original text information in a database and provides appropriate answers to questions from users. When a user enters a question on a smartphone or PC, the server searches the database and returns the relevant information.

[0523] Specific examples

[0524] Example 1: Sports day announcement

[0525] A user takes a photo of a sports day announcement printout and uploads it to the system. The server extracts the text information "The sports day will be held on July 15th at the school playground" from the image and summarizes it as "Sports day: July 15th" using a generative AI model. Then, an event called "Sports day" is registered in a calendar app for July 15th. When the user asks, "What time does the sports day start?", the server responds with, "The sports day starts at 9:00 AM."

[0526] Example 2: School lunch menu

[0527] A user takes a photo of a school lunch menu and uploads it to the system. The server extracts text information from the image, such as "July 1st: Curry rice, July 2nd: Fish meunière," and uses a generative AI model to summarize it as "July menu." The calendar is then populated with school lunch menus for each date. When a user asks, "What's for lunch on July 2nd?", the server responds, "The school lunch on July 2nd is fish meunière."

[0528] Prompt Sentence Examples

[0529] An example prompt for a generative AI model to summarize is:

[0530] Example prompt 1:

[0531] "Please summarize the following sentence concisely: 'The sports day will be held on July 15th at the school playground. It will start at 9:00 AM. Participants should assemble in their respective classes.'"

[0532] Example prompt 2:

[0533] "Based on the following information, please generate a summary for calendar registration: 'July 1st: Curry rice, July 2nd: Fish meunière, July 3rd: Hamburg steak'"

[0534] This system allows users to quickly understand the contents of printed materials, making it easier to manage schedules. It also allows users to quickly get answers to their questions, significantly reducing the burden of organizing printed materials.

[0535] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0536] Step 1:

[0537] The user uses a smartphone to take a photo of the document that the child has brought home, opens the dedicated app, selects the image, and presses the upload button, which sends the image data to the server.

[0538] Input: Image data of a document photographed with a smartphone

[0539] Output: Image data sent to the server

[0540] Specific behavior:

[0541] The user launches the camera app on their smartphone and takes a photo of the document.

[0542] The user opens the system's dedicated app and selects the photos they have taken.

[0543] The user presses the upload button in the app to send the image data to the server.

[0544] Step 2:

[0545] When the server receives the uploaded image data, it temporarily stores it, then uses OCR (optical character recognition) technology to extract character information from the image and convert it into text data.

[0546] Input: Image data uploaded to the server

[0547] Output: Extracted character information (text data)

[0548] Specific behavior:

[0549] The server temporarily stores the image data in a defined directory.

[0550] The server runs an OCR engine (e.g., Tesseract) to extract text information from the image.

[0551] The extracted character information is saved as a text file and passed on to the next process.

[0552] Step 3:

[0553] The server generates a prompt sentence based on the character information acquired by OCR and the generative AI model, and sends this prompt sentence to the generative AI model to obtain a summary.

[0554] Input: Character information obtained by OCR

[0555] Output: Generated prompt statement and summary information

[0556] Specific behavior:

[0557] The server generates a prompt based on the character information obtained by OCR.

[0558] The server sends prompt sentences to the generative AI model and obtains summaries.

[0559] The generated summary information is passed to the next process.

[0560] Step 4:

[0561] The server analyzes the summarized information to find relevant dates and event information and automatically registers them in the calendar application.

[0562] Input: Summarized information (text data)

[0563] Output: Event information registered in the calendar

[0564] Specific behavior:

[0565] The server parses the dates and event names contained in the summary.

[0566] The server uses a calendar API (e.g., Google Calendar API) to add the event information to the calendar.

[0567] Step 5:

[0568] The server stores summary information and the original text information in a database. When a user enters a question on a smartphone or computer, the server searches the database and provides the appropriate answer.

[0569] Input: Question from user (text data)

[0570] Output: Answer to the question (text data)

[0571] Specific behavior:

[0572] The server stores the OCR and summary information in a database.

[0573] Users enter their questions on their smartphones or computers.

[0574] The server searches a database and returns an answer based on the question entered.

[0575] (Application example 1)

[0576] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0577] In brick-and-mortar stores, it is difficult to efficiently manage paper media such as shift schedules and promotion information used by store staff in their daily work. This results in problems such as time-consuming sharing and updating of information, which reduces work efficiency. There is also a lack of means to quickly search for necessary information and obtain answers. There is a need for a system that can solve these problems.

[0578] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0579] In this invention, the server includes an uploading means for a user to photograph information on paper media as an image and convert the image into data, a character recognition means for the server to extract text information from the uploaded image, a summary generation means for the server to summarize the extracted text information, a calendar registration means for the server to automatically register the summary information and related date information in a calendar application, a question and answer means for a user to ask the system questions and the system answers the questions using the summary information or original data, and a store data linking means for linking the extracted text information and summary to a store management system. This enables store staff to efficiently digitize information on paper media and quickly search and manage it.

[0580] "User" refers to a person who uses the system to digitize and manage paper-based information.

[0581] "Paper media" refers to physical documents or books on which text, charts, etc. are printed.

[0582] The "uploading means" refers to a function that allows a user to take an image of information on a paper medium and send the image data to a server.

[0583] "Character recognition means" refers to the function by which the server extracts character information from uploaded images, and typically uses optical character recognition technology.

[0584] "Summary generation means" refers to the function by which the server succinctly summarizes the extracted text information, and typically uses a generative AI model.

[0585] The "calendar registration means" refers to a function that automatically registers summary information and related date information in a calendar application.

[0586] "Question and answer means" refers to a function that allows users to ask questions to the system and receive answers from summary information or original data.

[0587] "Store data linking means" refers to the function of associating extracted text information and summaries with a store management system.

[0588] "Server" refers to a computer device that processes, stores, and performs corresponding functions for the entire system.

[0589] "Physical store" refers to a store that exists in a physical location and is managed and operated by store staff.

[0590] The present invention is a system for efficiently managing paper-based information in a physical store, and is specifically implemented as follows.

[0591] System Configuration

[0592] This system automates the process by which users digitize paper media in their stores (e.g., shift schedules, promotion information) and register that information in a store management system. Specifically, it consists of devices such as smartphones and tablets, a server, and various software.

[0593] Hardware and Software

[0594] Hardware:

[0595] Smartphone or tablet

[0596] server

[0597] software:

[0598] Image processing library (OpenCV)

[0599] Optical Character Recognition Library (Tesseract-OCR)

[0600] Generative AI model (HuggingFace Transformers T5 model)

[0601] Calendar event generation library (icalendar)

[0602] Operating procedure

[0603] 1. The user photographs a paper medium: The user uses a smartphone or tablet to take a picture of a paper medium such as a shift schedule or promotional information.

[0604] 2. Uploading image data: The captured image data is uploaded to the server via a dedicated application.

[0605] 3. Character recognition by the server: The server preprocesses the received image using an image processing library (OpenCV) and extracts character information using an optical character recognition library (Tesseract-OCR).

[0606] 4. Summary generation: The extracted text information is summarized using a generative AI model (e.g., the T5 model from HuggingFace Transformers).

[0607] 5. Calendar Registration: The summary information and associated date information are automatically registered in the calendar application using the calendar event generation library (icalendar).

[0608] 6. Linkage to store management system: The extracted text information and summaries are linked to the store management system and reflected in staff work shifts and promotion schedules.

[0609] 7. Question and Answer System: Users can input questions into the system and the server will provide answers to the questions from summary information and original data.

[0610] Specific examples

[0611] Example 1: Managing shift schedules

[0612] A user takes a photo of a shift schedule and uploads the image to the server. The server extracts text information from the image, such as "July 1st: Person A is on duty, July 2nd: Person B is on duty," and summarizes it as the "July shift schedule" using a generative AI model. Shift information for each date is registered on the calendar. When a user asks, "Who is on the July 2nd shift?" the server responds, "Person B is on the July 2nd shift."

[0613] Example prompt sentence:

[0614] What's the next promotion?

[0615] Example 2: Managing promotion information

[0616] A user takes a photo of a printed copy of promotional information and uploads the image to a server. The server extracts text information from the image, such as "10% off sale on January 10th," and summarizes it using a generative AI model. An event called "Promotion: 10% off sale (January 10th)" is then registered in a calendar app. When the user asks, "What's the next promotion?", the server responds, "The next promotion is a 10% off sale on January 10th."

[0617] This system allows staff at physical stores to efficiently digitize paper-based information, enabling smooth business management.

[0618] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0619] Step 1:

[0620] Users take a photo of paper-based information (such as shift schedules or promotional information) using a smartphone or tablet. This generates image data, which is then uploaded to a server via a dedicated application. The input is the captured image data, and the output is the uploaded image data.

[0621] Step 2:

[0622] The server receives the uploaded image data and temporarily stores it. This image data is prepared for the next processing. The input is the uploaded image data, and the output is the stored image data.

[0623] Step 3:

[0624] The server preprocesses the received image data using an image processing library (OpenCV). Specifically, it converts the image to grayscale and performs noise reduction. The input is the stored image data, and the output is the preprocessed image data.

[0625] Step 4:

[0626] The server extracts character information from the preprocessed image using an optical character recognition library (Tesseract-OCR). The input is the preprocessed image data, and the output is text data as character information.

[0627] Step 5:

[0628] The server uses a generative AI model (e.g., the T5 model from HuggingFace Transformers) to summarize the extracted text information. The input is text data as text information, and the output is summarized information.

[0629] Step 6:

[0630] The server uses a calendar event generation library (icalendar) to register the summary information and associated date information to the calendar application. The input is the summary information and date information, and the output is the event registered in the calendar.

[0631] Step 7:

[0632] The server associates the extracted text and summary information with the store management system, which updates shift information and promotion schedules. The input is text and summary information, and the output is updated store management data.

[0633] Step 8:

[0634] The user inputs a question to the system. The server searches for an appropriate answer from the summary information and the original data based on the question and provides it to the user. The input is the user's question, and the output is the answer to the question.

[0635] This allows users to efficiently digitize paper-based information, making it easier to manage information within the store.

[0636] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0637] This invention is a system for efficiently managing a large amount of printed materials distributed by schools, and further has a function for recognizing the user's emotions and adjusting the response accordingly. Specific embodiments of the invention are described below.

[0638] Overall system overview

[0639] This system automates a series of processes: uploading images of prints taken by users, extracting text information from the images to generate summaries, registering them in a calendar application, and answering questions. It also incorporates an emotion engine that recognizes emotions from questions and messages entered by users and adjusts responses accordingly.

[0640] Program processing

[0641] 1. Taking a photo and uploading it

[0642] The user takes a photo of the printouts their child brings home using their smartphone. After taking the photo, the user uploads the image to the server using a dedicated app for the system. By pressing the upload button, the image data is sent to the server.

[0643] 2. Image Processing and Character Recognition

[0644] The server temporarily stores the received image for processing. The server then uses OCR (optical character recognition) technology to recognize the characters in the image and convert them into text data. This process digitizes the contents of the paper printout.

[0645] 3. Summary of Contents

[0646] The server summarizes the text information extracted by OCR using a generative model (e.g., GPT). This summary generation method produces a short and concise summary of the long printed content.

[0647] 4. Register with your calendar app

[0648] The server analyzes the summarized information and identifies relevant dates and events. For example, if the content says "The sports day will be held on the school grounds on July 15th," the date "July 15th" is identified and an event called "Sports Day" is registered on the calendar.

[0649] 5. Question and Answer System

[0650] The server stores the summary and the original text information in a database. When a user enters a question into the system using a smartphone or PC, the server searches the database for an answer to that question. For example, in response to a question such as "Where will the sports day be held?", the server will return an answer such as "The sports day will be held in the school's athletic field."

[0651] 6. Leveraging Emotional Engines

[0652] The server analyzes the questions and messages entered by the user and uses an emotion engine to recognize the user's emotional state. For example, if the user enters "It's too late! Tell me quickly!", the emotion engine will determine that the user is irritated.

[0653] If the emotion engine recognizes the user's emotion, the server will adjust the content and tone of the response, for example, responding to an annoyed user with a polite and prompt "I'm sorry, let me check again."

[0654] Furthermore, a dashboard can be provided to visually display the emotional state recognized by the emotion engine, allowing users to monitor changes in their emotions in real time and respond appropriately.

[0655] Specific examples

[0656] Example 1: Sports day announcement

[0657] A user takes a photo of a sports day announcement printout and uploads it to the system. The server extracts the text information "The sports day will be held on July 15th at the school playground" from the image and summarizes it as "Sports day: July 15th" using a summary generation model. An event called "Sports day" is then registered in the calendar app for July 15th. When the user asks, "What time does the sports day start?", the server replies, "The sports day starts at 9:00 AM." If the user impatiently asks, "Why is it so late?", the server responds politely with, "Sorry, I'll check right away."

[0658] Example 2: School lunch menu

[0659] A user takes a photo of a school lunch menu and uploads it to the system. The server extracts text information from the image, such as "July 1st: Curry rice, July 2nd: Fish meunière," and summarizes it as "July menu" using a summary generation model. The calendar registers the school lunch menu for each date. If a user asks, "What's for lunch on July 2nd?", the server responds, "The school lunch on July 2nd is fish meunière." If the user complains that the server is slow to respond, the server responds, "Sorry for the wait, we'll get back to you right away."

[0660] In this way, by utilizing the emotion engine, this system can efficiently summarize, manage, and answer questions about information while responding appropriately to the user's emotions.

[0661] The processing flow will be explained below.

[0662] Step 1:

[0663] The user takes a photo of the print that the child brought home using a smartphone terminal.

[0664] Step 2:

[0665] The images taken by the terminal are imported into the "School Print Summary System" application.

[0666] Step 3:

[0667] The user presses the "Upload" button in the app to send the image data to the server.

[0668] Step 4:

[0669] The server receives the uploaded image data and temporarily stores it.

[0670] Step 5:

[0671] The server invokes the OCR engine to extract text information from the stored image data, specifically recognizing text areas within the image and converting them into text data.

[0672] Step 6:

[0673] The server uses a generative AI model (e.g., GPT) to summarize the extracted text information. Specifically, it extracts key points from long text data and generates a concise summary.

[0674] Step 7:

[0675] The server parses the summarized information to identify relevant date and event information, specifically identifying dates and event names within the text.

[0676] Step 8:

[0677] The server calls the calendar application's API and automatically registers the specified events and dates in the calendar. For example, using the Google Calendar API, it registers "Sports Day" on July 15th.

[0678] Step 9:

[0679] The server stores the summary information and the original text in a database, allowing for later question and answer sessions.

[0680] Step 10:

[0681] A user opens a chat window within the system and types a question, for example, "Where is the sports day being held?"

[0682] Step 11:

[0683] The server receives the user's question, searches for relevant information in the database, and generates the best answer based on the summary information and the original text information.

[0684] Step 12:

[0685] The server sends the generated answer to the user, for example, "The sports day will be held on the school grounds."

[0686] Step 13:

[0687] The server analyzes the user's questions and messages and uses an emotion engine to recognize the user's emotional state. For example, if the user types "You're late! Tell me quickly!", the server detects the emotion of irritation.

[0688] Step 14:

[0689] The server adjusts the content and tone of the response depending on the emotion recognized by the emotion engine. If it detects irritation, it will respond politely and promptly, saying something like, "I'm sorry, we'll look into it right away."

[0690] Step 15:

[0691] The server reflects the user's emotions recognized by the emotion engine on a visual dashboard, which is used by system administrators to monitor the user's emotional state in real time.

[0692] The system operates according to these steps, allowing users to efficiently manage print content and quickly obtain the information they need. It also provides a better user experience by responding according to the user's emotions.

[0693] Example 2

[0694] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0695] Conventional print management systems have difficulty efficiently digitizing large amounts of paper media and managing summaries and schedules. Responses to user questions are also limited to standardized content, and there are also issues with not being able to recognize user emotions and respond flexibly.

[0696] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0697] In this invention, the server includes uploading means for a user to take an image of a recording medium and digitize the image, character recognition means for the server to extract character information from the uploaded image, summary generation means for the server to summarize the extracted character information, calendar registration means for the server to automatically register the summary information and related date information in a time management application, question and answering means for the user to ask a question to the system and to answer the question from the summary information or original data, emotion recognition means for the server to analyze the question or message entered by the user and recognize the user's emotional state using an emotion engine, and tone adjustment means for adjusting the content and tone of the response based on the recognition result by the emotion recognition means. This enables efficient digitization, summarization, automatic schedule registration, and flexible response based on emotions.

[0698] The "uploading means" is a means by which a user takes an image of a recording medium and transmits the image as data to a server.

[0699] "Character Recognition Means" refers to the techniques or tools used by the server to extract text information from uploaded images.

[0700] "Summary generation" refers to a generative model or other technology that allows the server to summarize extracted textual information in a concise, short format.

[0701] The "calendar registration means" is a means by which the server automatically registers summary information and related date information in the time management application.

[0702] The "question and answer means" is a means by which a user can ask a question to the system and have the system provide an answer based on the summary information or the original data in response to the question.

[0703] "Emotion recognition means" refers to an engine or technology that allows the server to analyze questions and messages entered by the user and recognize the user's emotional state.

[0704] The "tone adjustment means" is a means by which the server responds to the user by adjusting the content and tone of the response based on the recognition result by the emotion recognition means.

[0705] The present invention is a system for efficiently managing a large number of printed materials distributed by schools, recognizing the emotions of users, and adjusting responses accordingly. Specific embodiments of this system are described below.

[0706] Hardware / Software used

[0707] 1. Smartphone device:

[0708] This is a device used by users to take photos of prints and upload the images to a server using a dedicated app.

[0709] 2. Server:

[0710] A computer system that performs a series of processes such as image processing, character recognition, summary generation, calendar entry, and emotion recognition. It also includes a database.

[0711] 3. OCR (Optical Character Recognition) technology:

[0712] This is a technology for extracting text information from images. Specifically, Tesseract OCR is used.

[0713] 4. Generative AI Models:

[0714] This is a generative model for summarizing text information. For example, GPT-4 is used.

[0715] 5. Calendar application:

[0716] It is a time management tool that registers summary information and date information. It uses APIs such as Google Calendar.

[0717] 6. Emotion Recognition Engine:

[0718] This is a technology for analyzing user emotions, and uses sentiment analysis APIs, etc.

[0719] Specific examples

[0720] Example 1: Sports day announcement

[0721] A user takes a photo of a sports day announcement printout and uploads it to the server using a dedicated app. The server performs OCR processing on the received image and extracts the text information, such as "The sports day will be held on July 15th at the school's athletic field." The server then uses a generative AI model to summarize the information as "Sports day: July 15th." The summary information is then registered on the calendar as an event called "Sports day" using the Google Calendar API.

[0722] When a user asks, "What time does the sports day start?", the server searches the database for relevant information and replies, "The sports day starts at 9 a.m." If the user asks impatiently, "Why is it so late?", the emotion recognition engine will recognize the impatience and the server will respond flexibly, saying, "I'm sorry, I'll check right away."

[0723] Example 2: School lunch menu

[0724] The user takes a photo of the school lunch menu and uploads it to the server using a dedicated app. The server performs OCR on the image and extracts the text information "July 1st: Curry rice, July 2nd: Fish meunière." A generative AI model is used to summarize it as "July menu." The school lunch menu for each date is then registered on the calendar.

[0725] When a user asks, "What's for lunch on July 2nd?", the server responds, "The school lunch on July 2nd is fish meunière." If the user expresses dissatisfaction by saying, "You're taking so long to respond!", the emotion recognition engine recognizes the dissatisfaction, and the server responds, "Sorry to keep you waiting, we'll get back to you right away."

[0726] Prompt Sentence Examples

[0727] Example prompt 1:

[0728] "This is a printout announcing the sports day. Please summarize the following information and add it to your calendar. Content: 'The sports day will be held on July 15th at the school playground.'"

[0729] Example prompt 2:

[0730] "I have the school lunch menu in front of me. Please register the following school lunch menu items on your calendar. Contents: July 1st: Curry rice, July 2nd: Fish meunière."

[0731] As described above, this system efficiently digitizes, summarizes, and manages schedules based on user-taken print images. It also recognizes user emotions and responds flexibly accordingly. In this way, advanced OCR technology, generative AI models, and emotion recognition engines work together to improve the effectiveness of the entire system.

[0732] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0733] Step 1:

[0734] Taking photos and uploading

[0735] A user uses a smartphone device to take a photo of a printout that a child has brought home, for example, a printout listing a school event schedule.

[0736] Input: Printed image on paper

[0737] Output: Digital image files (JPEG, PNG, etc.)

[0738] Specific operation: The user launches the camera app on their smartphone, adjusts the screen so that the entire print fits on the screen, and takes a picture. After taking the picture, the image file is saved on the device.

[0739] Step 2:

[0740] Upload

[0741] The user launches the system's dedicated app and uploads the captured image to the server. Specifically, the user taps the "upload" button in the app, and the image is sent to the server.

[0742] Input: Digital image file

[0743] Output: Temporary file on the server

[0744] Specific operation: When the user taps the "Upload" button in the app, the selected image is sent to the server, where it is temporarily stored.

[0745] Step 3:

[0746] Image processing and character recognition

[0747] The server processes the uploaded image using OCR (Optical Character Recognition) technology to extract the text information from the image. Specifically, it uses tools such as Tesseract OCR to generate text data from the image.

[0748] Input: Temporary file on the server (digital image)

[0749] Output: Extracted character information (text data)

[0750] Specific operation: The server reads the saved image file and applies OCR technology to extract text information. The extracted text data is stored on the server.

[0751] Step 4:

[0752] Summary of contents

[0753] The server summarizes the text information extracted by OCR using a generative model (e.g., GPT-4). It provides a prompt such as "Please summarize the following content:"

[0754] Input: Extracted text information

[0755] Output: Summarized text information

[0756] Specific operation: The server inputs the extracted text information into the generative AI model and processes it along with the prompt "Please summarize the following content:" The generated summary is stored on the server.

[0757] Step 5:

[0758] Registering with a calendar app

[0759] The server identifies events and date information from the summarized information and automatically registers them in the calendar using an API such as Google Calendar.

[0760] Input: Summarized text information

[0761] Output: Event information registered in the calendar app

[0762] Specific operation: The server extracts the date, time, and event name from the summary text and registers the identified information in the calendar via the Google Calendar API. For example, "Sports Day: July 15th" is added to the calendar.

[0763] Step 6:

[0764] Q&A System

[0765] Users enter questions into the system using their smartphones or computers, and the server searches a database for answers to those questions and provides them.

[0766] Input: User question (text)

[0767] Output: Answer from the server (text)

[0768] Specific operation: When a user enters a question text through an app or web interface, the server searches the database for relevant information, generates an appropriate answer, and returns it to the user. For example, in response to the question, "What time does the sports day start?", the server responds with, "The sports day starts at 9:00 AM."

[0769] Step 7:

[0770] Utilizing the Emotion Engine

[0771] The server analyzes the questions and messages entered by the user, uses an emotion engine to recognize the user's emotional state, and adjusts the content and tone of the response based on the emotion recognition results.

[0772] Input: User question or message (text)

[0773] Output: Emotion recognition results and tailored responses (text)

[0774] What it does: The server extracts emotions from the user's text through an emotion analysis API, and if it detects irritation or frustration, it responds with a tailored response such as "Sorry, we'll take a look at it right away."

[0775] Emotional states are displayed in real time on a dashboard.

[0776] (Application example 2)

[0777] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0778] Conventional document management systems have difficulty efficiently digitizing and managing large volumes of documents, making the management of factory instructions and manuals particularly cumbersome. Furthermore, they do not respond to the emotions of operators, which can lead to problems such as reduced production efficiency and increased likelihood of work errors. The objective of this invention is to solve these problems by providing a system that responds appropriately to emotions while improving operator efficiency.

[0779] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: uploading means for a user to photograph an image of a document and digitize the image; character recognition means for the server to extract text information from the uploaded image; summary generation means for the server to summarize the extracted text information; calendar registration means for the server to automatically register the summary information and related date information in a calendar application; question and answer means for a user to ask the system questions and the system to answer the questions using the summary information or the original data; emotion analysis means for recognizing the user's emotional state and adjusting the response; means for photographing an image of a document using a photographing device attached to the item processing device; and transmission means for transmitting the photographed image to the processing device. This not only improves the efficiency of document management in the factory but also enables appropriate responses according to the operator's emotions.

[0780] The "uploading means" is a means for converting images taken by a user into data and transmitting the data to a server.

[0781] "Character recognition means" refers to a technology that extracts character information from uploaded images, and refers to optical character recognition technology.

[0782] The "summary generation means" is a technique for concisely summarizing extracted text information using a generative model.

[0783] The "calendar registration means" is a technique for automatically registering summary information and related date information in a calendar application.

[0784] "Question and answering means" is a technology that provides answers to questions posed by users to the system from summary information or original data.

[0785] "Emotion analysis means" is a technology that recognizes the emotional state of a user from questions or messages entered by the user and adjusts the response accordingly.

[0786] "Image capture device" refers to a device attached to an article processing device that captures an image of a document.

[0787] The "transmission means" is a means for transmitting the captured image to the processing device.

[0788] "Image processing means" refers to a series of technologies that analyze uploaded images and convert them into data.

[0789] The "information management system" is a series of systems that upload users' images, extract information from those images, summarize it, register it on a calendar, answer users' questions, and analyze users' emotions to adjust the response.

[0790] The following describes an embodiment of the present invention: This system uses smart glasses to capture images of documents and instructions used in a factory, digitize them, and manage them.

[0791] System Overview

[0792] An operator in a factory wears smart glasses and takes a photo of an instruction manual or instruction sheet. The captured image is processed by a dedicated application in the smart glasses and then uploaded to a server. The server extracts text information from the image, summarizes it, and registers it in a task management system. It also responds to the operator's questions and adjusts its response according to the operator's emotional state.

[0793] Hardware Configuration

[0794] Smart glasses: Worn by operators in the factory, they take photos of instructions and manuals.

[0795] Camera: Built into the smart glasses and equipped with a high-resolution camera.

[0796] Cloud server: Performs image processing, character recognition, summary generation, calendar registration, and sentiment analysis.

[0797] Task Management System: Functions as a calendar application, managing summary information and related date information.

[0798] Software Configuration

[0799] OCR software: Uses software such as Google Cloud Vision to extract text information from uploaded images.

[0800] Generative AI model: Summarize text information using OpenAI's GPT model.

[0801] Sentiment analysis engine: Using IBM Watson NLU and other technologies, it recognizes the emotional state of the user from their questions and messages and adjusts the response accordingly.

[0802] Dedicated application: Installed on smart glasses, it provides functions for taking photos, uploading images, and answering questions.

[0803] Processing flow

[0804] The server includes an uploading means for a user to take an image of a document and convert the image into data, a character recognition means for the server to extract character information from the uploaded image, a summary generation means for the server to summarize the extracted character information, a calendar registration means for the server to automatically register the summary information and related date information in a calendar application, a question and answer means for a user to ask a question to the system and the question is answered from the summary information or original data, an emotion analysis means for recognizing the emotional state of the user and adjusting the response, a means for taking an image of a document using a camera attached to the item processing device, and a transmission means for transmitting the taken image to the processing device.

[0805] Specific examples

[0806] For example, when an operator takes a photo of an instruction sheet using smart glasses, the image is first saved in a dedicated app on the smart glasses and then uploaded to a server. The server uses Google Cloud Vision for OCR processing to extract text information. The text is then summarized using OpenAI's GPT model, and the results are registered in the factory's task management system. When an operator asks, "What's the next task?", the Q&A system provides an answer based on the summary information and the original data. If the operator asks impatiently, "Why is it so slow?", IBM Watson NLU performs sentiment analysis and provides a polite response such as, "Sorry, we'll get back to you right away."

[0807] Prompt Sentence Examples

[0808] Summarize the following text:

[0809] The sports day will be held on July 15th at the school playground.

[0810] summary:

[0811] In this way, the invention makes it possible to improve the efficiency of document management within a factory and also to respond appropriately to the emotions of operators.

[0812] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0813] Process step description

[0814] (Step 1: User takes a photo of the document)

[0815] A user puts on the smart glasses and takes a photo of an instruction manual or instruction sheet. The input is the photographed paper document, and the output is a digital image file. Specifically, the glasses use a high-resolution camera to capture a clear image.

[0816] (Step 2: Save the image with the smart glasses)

[0817] The digital image file is stored in a dedicated application in the smart glasses. The input is the captured digital image, and the output is the image file stored in the smart glasses' storage. The specific operation is to perform the image file storage process.

[0818] (Step 3: Upload the image to the server)

[0819] The saved image files are uploaded to a server by a dedicated application in the smart glasses. The input is the image file saved in the smart glasses' storage, and the output is the image data sent to the server. Specifically, the data is sent via an internet connection.

[0820] (Step 4: Image processing and OCR on the server)

[0821] The server uses OCR software such as Google Cloud Vision to extract text information from uploaded images. The input is the image data uploaded to the server, and the output is text data with the text information. Specifically, OCR processing is performed to recognize the characters in the image.

[0822] (Step 5: Summary generation)

[0823] The server uses the character information extracted by OCR to generate a summary using OpenAI's GPT model. The input is text data extracted by OCR, and the output is a summarized text. The specific operation involves natural language processing using a generative AI model.

[0824] (Step 6: Register with your calendar application)

[0825] The server analyzes the summarized text information and related date information and automatically registers it in the factory's task management system. The input is the summarized text data, and the output is event information registered in a calendar application. Specific operations include text analysis and database registration.

[0826] (Step 7: Question and Answer System)

[0827] A user inputs a question into smart glasses or other devices, and the question is sent to a server. The server uses a question-and-answer system to generate an answer from summary information and the original data. The input is the user's question, and the output is the generated answer. Specifically, the system uses natural language processing technology to analyze the question and generate the answer.

[0828] (Step 8: Sentiment analysis and response adjustment)

[0829] The server analyzes the questions and messages entered by the user and uses an emotion analysis engine such as IBM Watson NLU to recognize the user's emotional state and adjust the response. The input is the user's question or message, and the output is a response that adjusts the emotional state and response. The specific operation involves emotion analysis and adjusting the response based on that.

[0830] This system not only improves the efficiency of document management within the factory, but also enables appropriate responses based on the emotions of operators.

[0831] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0832] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0833] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0834] [Third embodiment]

[0835] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0836] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0837] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0838] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0839] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0840] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0841] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0842] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0843] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0844] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0845] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0846] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0847] This invention is a system for efficiently managing a large amount of printed materials distributed by a school, and is specifically implemented as follows.

[0848] Overall system overview

[0849] The system automates a series of processes: users upload images of prints they have taken, extract text information from the images to generate summaries, register them in a calendar application, and then participate in a Q&A session.

[0850] Program processing

[0851] 1. Taking a photo and uploading it

[0852] The user takes a photo of the printouts their child brings home using their smartphone. After taking the photo, the user uploads the image to the server using a dedicated app for the system. By pressing the upload button, the image data is sent to the server.

[0853] 2. Image Processing and Character Recognition

[0854] The server temporarily stores the received image for processing. The server then uses OCR (optical character recognition) technology to recognize the characters in the image and convert them into text data. This process digitizes the contents of the paper printout.

[0855] 3. Summary of Contents

[0856] The server summarizes the text information extracted by OCR using a generative model (e.g., GPT). This summary generation method produces a short and concise summary of the long printed content.

[0857] 4. Register with your calendar app

[0858] The server analyzes the summarized information and automatically registers relevant dates and event information in a calendar application. For example, if the content says "The sports day will be held on the school playground on July 15th," the date "July 15th" will be identified and registered as an event called "Sports Day" in the calendar.

[0859] 5. Question and Answer System

[0860] The server stores the summary and the original text information in a database. When a user enters a question into the system using a smartphone or PC, the server searches the database for an answer to that question. For example, in response to a question such as "Where will the sports day be held?", the server will return an answer such as "The sports day will be held in the school's athletic field."

[0861] Specific examples

[0862] Example 1: Sports day announcement

[0863] A user takes a photo of a notice printout for a sports day and uploads it to the system. The server extracts the text information "The sports day will be held on July 15th at the school playground" from the image and summarizes it as "Sports day: July 15th" using a summary generation model. After that, an event called "Sports day" is registered in the calendar app for July 15th. When the user asks "What time does the sports day start?", the server replies "The sports day starts at 9:00 AM."

[0864] Example 2: School lunch menu

[0865] A user takes a photo of a school lunch menu and uploads it to the system. The server extracts text information from the image, such as "July 1st: curry rice, July 2nd: fish meunière," and summarizes it as "July menu" using a summary generation model. The calendar is registered with the school lunch menu for each date. When a user asks, "What's for lunch on July 2nd?", the server responds, "The school lunch on July 2nd is fish meunière."

[0866] This system allows users to quickly understand the contents of printed materials, making it easier to manage schedules. It also allows users to quickly get answers to their questions, significantly reducing the burden of organizing printed materials.

[0867] The processing flow will be explained below.

[0868] Step 1:

[0869] The user takes a photo of the print that the child brought home using a smartphone terminal.

[0870] Step 2:

[0871] The images taken by the terminal are imported into the "School Print Summary System" application.

[0872] Step 3:

[0873] The user presses the "Upload" button in the app to send the image data to the server.

[0874] Step 4:

[0875] The server receives the uploaded image data and temporarily stores it.

[0876] Step 5:

[0877] The server invokes the OCR engine to extract text information from the stored image data, specifically recognizing text areas within the image and converting them into text data.

[0878] Step 6:

[0879] The server uses a generative AI model (e.g., GPT) to summarize the extracted text information. Specifically, it extracts key points from long text data and generates a concise summary.

[0880] Step 7:

[0881] The server parses the summarized information to identify relevant date and event information, specifically identifying dates and event names within the text.

[0882] Step 8:

[0883] The server calls the calendar application's API and automatically registers the specified events and dates in the calendar. For example, using the Google Calendar API, it registers "Sports Day" on July 15th.

[0884] Step 9:

[0885] The server stores the summary information and the original text in a database, allowing for later question and answer sessions.

[0886] Step 10:

[0887] A user opens a chat window within the system and types a question, for example, "Where is the sports day being held?"

[0888] Step 11:

[0889] The server receives the user's question, searches for relevant information in the database, and generates the best answer based on the summary information and the original text information.

[0890] Step 12:

[0891] The server sends the generated answer to the user, for example, "The sports day will be held on the school grounds."

[0892] The system operates according to these steps, allowing users to efficiently manage print content and quickly obtain the information they need.

[0893] Example 1

[0894] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0895] Conventional document management systems lacked the means to efficiently manage the large number of printed materials distributed by schools, requiring users to manually check the contents and manage schedules. Furthermore, they were unable to effectively combine question and answer functions, making it difficult to quickly obtain information. This resulted in significant waste of time and effort.

[0896] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0897] In this invention, the server includes a prompt generation means for generating a prompt sentence based on character information acquired by OCR using a generative AI model, a summary generation means for sending the generated prompt sentence to the generative AI model and generating a summary, and a calendar registration means for automatically registering the summary information and related date information in a calendar application, thereby enabling a user to quickly understand the contents of a document, efficiently manage their schedule, and quickly obtain necessary information.

[0898] "User" refers to a person who uses the system to take images of documents and review and manage information.

[0899] "Documents" refers to physical documents such as handouts and notices distributed by the school.

[0900] "Image" refers to the digital photograph obtained when a user photographs a document.

[0901] "Server" refers to the central system that processes images uploaded by users and performs character recognition, summary generation, calendar registration, and question and answering.

[0902] "Uploading means" refers to a function or device for sending images taken by a user to a server.

[0903] "Character recognition means" refers to a function or device that extracts character information from uploaded images.

[0904] "Summary generation means" refers to a function or device that briefly summarizes extracted text information.

[0905] "Calendar registration means" refers to a function or device that automatically registers summarized information and related date information in a calendar application.

[0906] "Question and answering means" refers to a function or device that answers questions posed by users to the system using summary information or original data.

[0907] "OCR" stands for optical character recognition technology, which refers to the technology of extracting text information from images.

[0908] "Prompt generation means" refers to a function or device that generates a prompt sentence based on character information obtained by OCR.

[0909] A "generative AI model" refers to an artificial intelligence model that performs natural language processing and enables summarization and question-and-answering.

[0910] This invention is a system for efficiently managing large volumes of documents distributed by schools. This system automates a series of processes: users take photos of documents, upload them to a server, extract text information using OCR technology, generate summaries using a generative AI model, register the summaries in a calendar application, and then handle questions and answers.

[0911] Hardware and software used

[0912] Terminal

[0913] Users use their smartphones to take pictures of documents and then upload the images to the server using a dedicated application for the system.

[0914] server

[0915] The server implements the following functions:

[0916] 1. Character recognition means

[0917] The server uses OCR (Optical Character Recognition) technology, such as the Tesseract OCR library, to extract text information from the uploaded image.

[0918] 2. Prompt Generation Method

[0919] The server generates a prompt based on the text information obtained by OCR, which is then sent to a generative AI model to generate a summary.

[0920] 3. Summary generation means

[0921] The server summarizes text information using a generative AI model such as GPT-3. The server inputs the prompt sentence generated by the prompt generation means into the model and obtains the summary result.

[0922] 4. Calendar registration method

[0923] The server analyzes the summarized information to find relevant dates and events, and automatically registers them in a calendar application. For example, by using the Google Calendar API, extracted event information can be added to a calendar.

[0924] 5. Q&A tools

[0925] The server stores the summary information and the original text information in a database and provides appropriate answers to questions from users. When a user enters a question on a smartphone or PC, the server searches the database and returns the relevant information.

[0926] Specific examples

[0927] Example 1: Sports day announcement

[0928] A user takes a photo of a sports day announcement printout and uploads it to the system. The server extracts the text information "The sports day will be held on July 15th at the school playground" from the image and summarizes it as "Sports day: July 15th" using a generative AI model. Then, an event called "Sports day" is registered in a calendar app for July 15th. When the user asks, "What time does the sports day start?", the server responds with, "The sports day starts at 9:00 AM."

[0929] Example 2: School lunch menu

[0930] A user takes a photo of a school lunch menu and uploads it to the system. The server extracts text information from the image, such as "July 1st: Curry rice, July 2nd: Fish meunière," and uses a generative AI model to summarize it as "July menu." The calendar is then populated with school lunch menus for each date. When a user asks, "What's for lunch on July 2nd?", the server responds, "The school lunch on July 2nd is fish meunière."

[0931] Prompt Sentence Examples

[0932] An example prompt for a generative AI model to summarize is:

[0933] Example prompt 1:

[0934] "Please summarize the following sentence concisely: 'The sports day will be held on July 15th at the school playground. It will start at 9:00 AM. Participants should assemble in their respective classes.'"

[0935] Example prompt 2:

[0936] "Based on the following information, please generate a summary for calendar registration: 'July 1st: Curry rice, July 2nd: Fish meunière, July 3rd: Hamburg steak'"

[0937] This system allows users to quickly understand the contents of printed materials, making it easier to manage schedules. It also allows users to quickly get answers to their questions, significantly reducing the burden of organizing printed materials.

[0938] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0939] Step 1:

[0940] The user uses a smartphone to take a photo of the document that the child has brought home, opens the dedicated app, selects the image, and presses the upload button, which sends the image data to the server.

[0941] Input: Image data of a document photographed with a smartphone

[0942] Output: Image data sent to the server

[0943] Specific behavior:

[0944] The user launches the camera app on their smartphone and takes a photo of the document.

[0945] The user opens the system's dedicated app and selects the photos they have taken.

[0946] The user presses the upload button in the app to send the image data to the server.

[0947] Step 2:

[0948] When the server receives the uploaded image data, it temporarily stores it, then uses OCR (optical character recognition) technology to extract character information from the image and convert it into text data.

[0949] Input: Image data uploaded to the server

[0950] Output: Extracted character information (text data)

[0951] Specific behavior:

[0952] The server temporarily stores the image data in a defined directory.

[0953] The server runs an OCR engine (e.g., Tesseract) to extract text information from the image.

[0954] The extracted character information is saved as a text file and passed on to the next process.

[0955] Step 3:

[0956] The server generates a prompt sentence based on the character information acquired by OCR and the generative AI model, and sends this prompt sentence to the generative AI model to obtain a summary.

[0957] Input: Character information obtained by OCR

[0958] Output: Generated prompt statement and summary information

[0959] Specific behavior:

[0960] The server generates a prompt based on the character information obtained by OCR.

[0961] The server sends prompt sentences to the generative AI model and obtains summaries.

[0962] The generated summary information is passed to the next process.

[0963] Step 4:

[0964] The server analyzes the summarized information to find relevant dates and event information and automatically registers them in the calendar application.

[0965] Input: Summarized information (text data)

[0966] Output: Event information registered in the calendar

[0967] Specific behavior:

[0968] The server parses the dates and event names contained in the summary.

[0969] The server uses a calendar API (e.g., Google Calendar API) to add the event information to the calendar.

[0970] Step 5:

[0971] The server stores summary information and the original text information in a database. When a user enters a question on a smartphone or computer, the server searches the database and provides the appropriate answer.

[0972] Input: Question from user (text data)

[0973] Output: Answer to the question (text data)

[0974] Specific behavior:

[0975] The server stores the OCR and summary information in a database.

[0976] Users enter their questions on their smartphones or computers.

[0977] The server searches a database and returns an answer based on the question entered.

[0978] (Application example 1)

[0979] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0980] In brick-and-mortar stores, it is difficult to efficiently manage paper media such as shift schedules and promotion information used by store staff in their daily work. This results in problems such as time-consuming sharing and updating of information, which reduces work efficiency. There is also a lack of means to quickly search for necessary information and obtain answers. There is a need for a system that can solve these problems.

[0981] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0982] In this invention, the server includes an uploading means for a user to photograph information on paper media as an image and convert the image into data, a character recognition means for the server to extract text information from the uploaded image, a summary generation means for the server to summarize the extracted text information, a calendar registration means for the server to automatically register the summary information and related date information in a calendar application, a question and answer means for a user to ask the system questions and the system answers the questions using the summary information or original data, and a store data linking means for linking the extracted text information and summary to a store management system. This enables store staff to efficiently digitize information on paper media and quickly search and manage it.

[0983] "User" refers to a person who uses the system to digitize and manage paper-based information.

[0984] "Paper media" refers to physical documents or books on which text, charts, etc. are printed.

[0985] The "uploading means" refers to a function that allows a user to take an image of information on a paper medium and send the image data to a server.

[0986] "Character recognition means" refers to the function by which the server extracts character information from uploaded images, and typically uses optical character recognition technology.

[0987] "Summary generation means" refers to the function by which the server succinctly summarizes the extracted text information, and typically uses a generative AI model.

[0988] The "calendar registration means" refers to a function that automatically registers summary information and related date information in a calendar application.

[0989] "Question and answer means" refers to a function that allows users to ask questions to the system and receive answers from summary information or original data.

[0990] "Store data linking means" refers to the function of associating extracted text information and summaries with a store management system.

[0991] "Server" refers to a computer device that processes, stores, and performs corresponding functions for the entire system.

[0992] "Physical store" refers to a store that exists in a physical location and is managed and operated by store staff.

[0993] The present invention is a system for efficiently managing paper-based information in a physical store, and is specifically implemented as follows.

[0994] System Configuration

[0995] This system automates the process by which users digitize paper media in their stores (e.g., shift schedules, promotion information) and register that information in a store management system. Specifically, it consists of devices such as smartphones and tablets, a server, and various software.

[0996] Hardware and Software

[0997] Hardware:

[0998] Smartphone or tablet

[0999] server

[1000] software:

[1001] Image processing library (OpenCV)

[1002] Optical Character Recognition Library (Tesseract-OCR)

[1003] Generative AI model (HuggingFace Transformers T5 model)

[1004] Calendar event generation library (icalendar)

[1005] Operating procedure

[1006] 1. The user photographs a paper medium: The user uses a smartphone or tablet to take a picture of a paper medium such as a shift schedule or promotional information.

[1007] 2. Uploading image data: The captured image data is uploaded to the server via a dedicated application.

[1008] 3. Character recognition by the server: The server preprocesses the received image using an image processing library (OpenCV) and extracts character information using an optical character recognition library (Tesseract-OCR).

[1009] 4. Summary generation: The extracted text information is summarized using a generative AI model (e.g., the T5 model from HuggingFace Transformers).

[1010] 5. Calendar Registration: The summary information and associated date information are automatically registered in the calendar application using the calendar event generation library (icalendar).

[1011] 6. Linkage to store management system: The extracted text information and summaries are linked to the store management system and reflected in staff work shifts and promotion schedules.

[1012] 7. Question and Answer System: Users can input questions into the system and the server will provide answers to the questions from summary information and original data.

[1013] Specific examples

[1014] Example 1: Managing shift schedules

[1015] A user takes a photo of a shift schedule and uploads the image to the server. The server extracts text information from the image, such as "July 1st: Person A is on duty, July 2nd: Person B is on duty," and summarizes it as the "July shift schedule" using a generative AI model. Shift information for each date is registered on the calendar. When a user asks, "Who is on the July 2nd shift?" the server responds, "Person B is on the July 2nd shift."

[1016] Example prompt sentence:

[1017] What's the next promotion?

[1018] Example 2: Managing promotion information

[1019] A user takes a photo of a printed copy of promotional information and uploads the image to a server. The server extracts text information from the image, such as "10% off sale on January 10th," and summarizes it using a generative AI model. An event called "Promotion: 10% off sale (January 10th)" is then registered in a calendar app. When the user asks, "What's the next promotion?", the server responds, "The next promotion is a 10% off sale on January 10th."

[1020] This system allows staff at physical stores to efficiently digitize paper-based information, enabling smooth business management.

[1021] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1022] Step 1:

[1023] Users take a photo of paper-based information (such as shift schedules or promotional information) using a smartphone or tablet. This generates image data, which is then uploaded to a server via a dedicated application. The input is the captured image data, and the output is the uploaded image data.

[1024] Step 2:

[1025] The server receives the uploaded image data and temporarily stores it. This image data is prepared for the next processing. The input is the uploaded image data, and the output is the stored image data.

[1026] Step 3:

[1027] The server preprocesses the received image data using an image processing library (OpenCV). Specifically, it converts the image to grayscale and performs noise reduction. The input is the stored image data, and the output is the preprocessed image data.

[1028] Step 4:

[1029] The server extracts character information from the preprocessed image using an optical character recognition library (Tesseract-OCR). The input is the preprocessed image data, and the output is text data as character information.

[1030] Step 5:

[1031] The server uses a generative AI model (e.g., the T5 model from HuggingFace Transformers) to summarize the extracted text information. The input is text data as text information, and the output is summarized information.

[1032] Step 6:

[1033] The server uses a calendar event generation library (icalendar) to register the summary information and associated date information to the calendar application. The input is the summary information and date information, and the output is the event registered in the calendar.

[1034] Step 7:

[1035] The server associates the extracted text and summary information with the store management system, which updates shift information and promotion schedules. The input is text and summary information, and the output is updated store management data.

[1036] Step 8:

[1037] The user inputs a question to the system. The server searches for an appropriate answer from the summary information and the original data based on the question and provides it to the user. The input is the user's question, and the output is the answer to the question.

[1038] This allows users to efficiently digitize paper-based information, making it easier to manage information within the store.

[1039] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1040] This invention is a system for efficiently managing a large amount of printed materials distributed by schools, and further has a function for recognizing the user's emotions and adjusting the response accordingly. Specific embodiments of the invention are described below.

[1041] Overall system overview

[1042] This system automates a series of processes: uploading images of prints taken by users, extracting text information from the images to generate summaries, registering them in a calendar application, and answering questions. It also incorporates an emotion engine that recognizes emotions from questions and messages entered by users and adjusts responses accordingly.

[1043] Program processing

[1044] 1. Taking a photo and uploading it

[1045] The user takes a photo of the printouts their child brings home using their smartphone. After taking the photo, the user uploads the image to the server using a dedicated app for the system. By pressing the upload button, the image data is sent to the server.

[1046] 2. Image Processing and Character Recognition

[1047] The server temporarily stores the received image for processing. The server then uses OCR (optical character recognition) technology to recognize the characters in the image and convert them into text data. This process digitizes the contents of the paper printout.

[1048] 3. Summary of Contents

[1049] The server summarizes the text information extracted by OCR using a generative model (e.g., GPT). This summary generation method produces a short and concise summary of the long printed content.

[1050] 4. Register with your calendar app

[1051] The server analyzes the summarized information and identifies relevant dates and events. For example, if the content says "The sports day will be held on the school grounds on July 15th," the date "July 15th" is identified and an event called "Sports Day" is registered on the calendar.

[1052] 5. Question and Answer System

[1053] The server stores the summary and the original text information in a database. When a user enters a question into the system using a smartphone or PC, the server searches the database for an answer to that question. For example, in response to a question such as "Where will the sports day be held?", the server will return an answer such as "The sports day will be held in the school's athletic field."

[1054] 6. Leveraging Emotional Engines

[1055] The server analyzes the questions and messages entered by the user and uses an emotion engine to recognize the user's emotional state. For example, if the user enters "It's too late! Tell me quickly!", the emotion engine will determine that the user is irritated.

[1056] If the emotion engine recognizes the user's emotion, the server will adjust the content and tone of the response, for example, responding to an annoyed user with a polite and prompt "I'm sorry, let me check again."

[1057] Furthermore, a dashboard can be provided to visually display the emotional state recognized by the emotion engine, allowing users to monitor changes in their emotions in real time and respond appropriately.

[1058] Specific examples

[1059] Example 1: Sports day announcement

[1060] A user takes a photo of a sports day announcement printout and uploads it to the system. The server extracts the text information "The sports day will be held on July 15th at the school playground" from the image and summarizes it as "Sports day: July 15th" using a summary generation model. An event called "Sports day" is then registered in the calendar app for July 15th. When the user asks, "What time does the sports day start?", the server replies, "The sports day starts at 9:00 AM." If the user impatiently asks, "Why is it so late?", the server responds politely with, "Sorry, I'll check right away."

[1061] Example 2: School lunch menu

[1062] A user takes a photo of a school lunch menu and uploads it to the system. The server extracts text information from the image, such as "July 1st: Curry rice, July 2nd: Fish meunière," and summarizes it as "July menu" using a summary generation model. The calendar registers the school lunch menu for each date. If a user asks, "What's for lunch on July 2nd?", the server responds, "The school lunch on July 2nd is fish meunière." If the user complains that the server is slow to respond, the server responds, "Sorry for the wait, we'll get back to you right away."

[1063] In this way, by utilizing the emotion engine, this system can efficiently summarize, manage, and answer questions about information while responding appropriately to the user's emotions.

[1064] The processing flow will be explained below.

[1065] Step 1:

[1066] The user takes a photo of the print that the child brought home using a smartphone terminal.

[1067] Step 2:

[1068] The images taken by the terminal are imported into the "School Print Summary System" application.

[1069] Step 3:

[1070] The user presses the "Upload" button in the app to send the image data to the server.

[1071] Step 4:

[1072] The server receives the uploaded image data and temporarily stores it.

[1073] Step 5:

[1074] The server invokes the OCR engine to extract text information from the stored image data, specifically recognizing text areas within the image and converting them into text data.

[1075] Step 6:

[1076] The server uses a generative AI model (e.g., GPT) to summarize the extracted text information. Specifically, it extracts key points from long text data and generates a concise summary.

[1077] Step 7:

[1078] The server parses the summarized information to identify relevant date and event information, specifically identifying dates and event names within the text.

[1079] Step 8:

[1080] The server calls the calendar application's API and automatically registers the specified events and dates in the calendar. For example, using the Google Calendar API, it registers "Sports Day" on July 15th.

[1081] Step 9:

[1082] The server stores the summary information and the original text in a database, allowing for later question and answer sessions.

[1083] Step 10:

[1084] A user opens a chat window within the system and types a question, for example, "Where is the sports day being held?"

[1085] Step 11:

[1086] The server receives the user's question, searches for relevant information in the database, and generates the best answer based on the summary information and the original text information.

[1087] Step 12:

[1088] The server sends the generated answer to the user, for example, "The sports day will be held on the school grounds."

[1089] Step 13:

[1090] The server analyzes the user's questions and messages and uses an emotion engine to recognize the user's emotional state. For example, if the user types "You're late! Tell me quickly!", the server detects the emotion of irritation.

[1091] Step 14:

[1092] The server adjusts the content and tone of the response depending on the emotion recognized by the emotion engine. If it detects irritation, it will respond politely and promptly, saying something like, "I'm sorry, we'll look into it right away."

[1093] Step 15:

[1094] The server reflects the user's emotions recognized by the emotion engine on a visual dashboard, which is used by system administrators to monitor the user's emotional state in real time.

[1095] The system operates according to these steps, allowing users to efficiently manage print content and quickly obtain the information they need. It also provides a better user experience by responding according to the user's emotions.

[1096] Example 2

[1097] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1098] Conventional print management systems have difficulty efficiently digitizing large amounts of paper media and managing summaries and schedules. Responses to user questions are also limited to standardized content, and there are also issues with not being able to recognize user emotions and respond flexibly.

[1099] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1100] In this invention, the server includes uploading means for a user to take an image of a recording medium and digitize the image, character recognition means for the server to extract character information from the uploaded image, summary generation means for the server to summarize the extracted character information, calendar registration means for the server to automatically register the summary information and related date information in a time management application, question and answering means for the user to ask a question to the system and to answer the question from the summary information or original data, emotion recognition means for the server to analyze the question or message entered by the user and recognize the user's emotional state using an emotion engine, and tone adjustment means for adjusting the content and tone of the response based on the recognition result by the emotion recognition means. This enables efficient digitization, summarization, automatic schedule registration, and flexible response based on emotions.

[1101] The "uploading means" is a means by which a user takes an image of a recording medium and transmits the image as data to a server.

[1102] "Character Recognition Means" refers to the techniques or tools used by the server to extract text information from uploaded images.

[1103] "Summary generation" refers to a generative model or other technology that allows the server to summarize extracted textual information in a concise, short format.

[1104] The "calendar registration means" is a means by which the server automatically registers summary information and related date information in the time management application.

[1105] The "question and answer means" is a means by which a user can ask a question to the system and have the system provide an answer based on the summary information or the original data in response to the question.

[1106] "Emotion recognition means" refers to an engine or technology that allows the server to analyze questions and messages entered by the user and recognize the user's emotional state.

[1107] The "tone adjustment means" is a means by which the server responds to the user by adjusting the content and tone of the response based on the recognition result by the emotion recognition means.

[1108] The present invention is a system for efficiently managing a large number of printed materials distributed by schools, recognizing the emotions of users, and adjusting responses accordingly. Specific embodiments of this system are described below.

[1109] Hardware / Software used

[1110] 1. Smartphone device:

[1111] This is a device used by users to take photos of prints and upload the images to a server using a dedicated app.

[1112] 2. Server:

[1113] A computer system that performs a series of processes such as image processing, character recognition, summary generation, calendar entry, and emotion recognition. It also includes a database.

[1114] 3. OCR (Optical Character Recognition) technology:

[1115] This is a technology for extracting text information from images. Specifically, Tesseract OCR is used.

[1116] 4. Generative AI Models:

[1117] This is a generative model for summarizing text information. For example, GPT-4 is used.

[1118] 5. Calendar application:

[1119] It is a time management tool that registers summary information and date information. It uses APIs such as Google Calendar.

[1120] 6. Emotion Recognition Engine:

[1121] This is a technology for analyzing user emotions, and uses sentiment analysis APIs, etc.

[1122] Specific examples

[1123] Example 1: Sports day announcement

[1124] A user takes a photo of a sports day announcement printout and uploads it to the server using a dedicated app. The server performs OCR processing on the received image and extracts the text information, such as "The sports day will be held on July 15th at the school's athletic field." The server then uses a generative AI model to summarize the information as "Sports day: July 15th." The summary information is then registered on the calendar as an event called "Sports day" using the Google Calendar API.

[1125] When a user asks, "What time does the sports day start?", the server searches the database for relevant information and replies, "The sports day starts at 9 a.m." If the user asks impatiently, "Why is it so late?", the emotion recognition engine will recognize the impatience and the server will respond flexibly, saying, "I'm sorry, I'll check right away."

[1126] Example 2: School lunch menu

[1127] The user takes a photo of the school lunch menu and uploads it to the server using a dedicated app. The server performs OCR on the image and extracts the text information "July 1st: Curry rice, July 2nd: Fish meunière." A generative AI model is used to summarize it as "July menu." The school lunch menu for each date is then registered on the calendar.

[1128] When a user asks, "What's for lunch on July 2nd?", the server responds, "The school lunch on July 2nd is fish meunière." If the user expresses dissatisfaction by saying, "You're taking so long to respond!", the emotion recognition engine recognizes the dissatisfaction, and the server responds, "Sorry to keep you waiting, we'll get back to you right away."

[1129] Prompt Sentence Examples

[1130] Example prompt 1:

[1131] "This is a printout announcing the sports day. Please summarize the following information and add it to your calendar. Content: 'The sports day will be held on July 15th at the school playground.'"

[1132] Example prompt 2:

[1133] "I have the school lunch menu in front of me. Please register the following school lunch menu items on your calendar. Contents: July 1st: Curry rice, July 2nd: Fish meunière."

[1134] As described above, this system efficiently digitizes, summarizes, and manages schedules based on user-taken print images. It also recognizes user emotions and responds flexibly accordingly. In this way, advanced OCR technology, generative AI models, and emotion recognition engines work together to improve the effectiveness of the entire system.

[1135] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1136] Step 1:

[1137] Taking photos and uploading

[1138] A user uses a smartphone device to take a photo of a printout that a child has brought home, for example, a printout listing a school event schedule.

[1139] Input: Printed image on paper

[1140] Output: Digital image files (JPEG, PNG, etc.)

[1141] Specific operation: The user launches the camera app on their smartphone, adjusts the screen so that the entire print fits on the screen, and takes a picture. After taking the picture, the image file is saved on the device.

[1142] Step 2:

[1143] Upload

[1144] The user launches the system's dedicated app and uploads the captured image to the server. Specifically, the user taps the "upload" button in the app, and the image is sent to the server.

[1145] Input: Digital image file

[1146] Output: Temporary file on the server

[1147] Specific operation: When the user taps the "Upload" button in the app, the selected image is sent to the server, where it is temporarily stored.

[1148] Step 3:

[1149] Image processing and character recognition

[1150] The server processes the uploaded image using OCR (Optical Character Recognition) technology to extract the text information from the image. Specifically, it uses tools such as Tesseract OCR to generate text data from the image.

[1151] Input: Temporary file on the server (digital image)

[1152] Output: Extracted character information (text data)

[1153] Specific operation: The server reads the saved image file and applies OCR technology to extract text information. The extracted text data is stored on the server.

[1154] Step 4:

[1155] Summary of contents

[1156] The server summarizes the text information extracted by OCR using a generative model (e.g., GPT-4). It provides a prompt such as "Please summarize the following content:"

[1157] Input: Extracted text information

[1158] Output: Summarized text information

[1159] Specific operation: The server inputs the extracted text information into the generative AI model and processes it along with the prompt "Please summarize the following content:" The generated summary is stored on the server.

[1160] Step 5:

[1161] Registering with a calendar app

[1162] The server identifies events and date information from the summarized information and automatically registers them in the calendar using an API such as Google Calendar.

[1163] Input: Summarized text information

[1164] Output: Event information registered in the calendar app

[1165] Specific operation: The server extracts the date, time, and event name from the summary text and registers the identified information in the calendar via the Google Calendar API. For example, "Sports Day: July 15th" is added to the calendar.

[1166] Step 6:

[1167] Q&A System

[1168] Users enter questions into the system using their smartphones or computers, and the server searches a database for answers to those questions and provides them.

[1169] Input: User question (text)

[1170] Output: Answer from the server (text)

[1171] Specific operation: When a user enters a question text through an app or web interface, the server searches the database for relevant information, generates an appropriate answer, and returns it to the user. For example, in response to the question, "What time does the sports day start?", the server responds with, "The sports day starts at 9:00 AM."

[1172] Step 7:

[1173] Utilizing the Emotion Engine

[1174] The server analyzes the questions and messages entered by the user, uses an emotion engine to recognize the user's emotional state, and adjusts the content and tone of the response based on the emotion recognition results.

[1175] Input: User question or message (text)

[1176] Output: Emotion recognition results and tailored responses (text)

[1177] What it does: The server extracts emotions from the user's text through an emotion analysis API, and if it detects irritation or frustration, it responds with a tailored response such as "Sorry, we'll take a look at it right away."

[1178] Emotional states are displayed in real time on a dashboard.

[1179] (Application example 2)

[1180] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1181] Conventional document management systems have difficulty efficiently digitizing and managing large volumes of documents, making the management of factory instructions and manuals particularly cumbersome. Furthermore, they do not respond to the emotions of operators, which can lead to problems such as reduced production efficiency and increased likelihood of work errors. The objective of this invention is to solve these problems by providing a system that responds appropriately to emotions while improving operator efficiency.

[1182] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: uploading means for a user to photograph an image of a document and digitize the image; character recognition means for the server to extract text information from the uploaded image; summary generation means for the server to summarize the extracted text information; calendar registration means for the server to automatically register the summary information and related date information in a calendar application; question and answer means for a user to ask the system questions and the system to answer the questions using the summary information or the original data; emotion analysis means for recognizing the user's emotional state and adjusting the response; means for photographing an image of a document using a photographing device attached to the item processing device; and transmission means for transmitting the photographed image to the processing device. This not only improves the efficiency of document management in the factory but also enables appropriate responses according to the operator's emotions.

[1183] The "uploading means" is a means for converting images taken by a user into data and transmitting the data to a server.

[1184] "Character recognition means" refers to a technology that extracts character information from uploaded images, and refers to optical character recognition technology.

[1185] The "summary generation means" is a technique for concisely summarizing extracted text information using a generative model.

[1186] The "calendar registration means" is a technique for automatically registering summary information and related date information in a calendar application.

[1187] "Question and answering means" is a technology that provides answers to questions posed by users to the system from summary information or original data.

[1188] "Emotion analysis means" is a technology that recognizes the emotional state of a user from questions or messages entered by the user and adjusts the response accordingly.

[1189] "Image capture device" refers to a device attached to an article processing device that captures an image of a document.

[1190] The "transmission means" is a means for transmitting the captured image to the processing device.

[1191] "Image processing means" refers to a series of technologies that analyze uploaded images and convert them into data.

[1192] The "information management system" is a series of systems that upload users' images, extract information from those images, summarize it, register it on a calendar, answer users' questions, and analyze users' emotions to adjust the response.

[1193] The following describes an embodiment of the present invention: This system uses smart glasses to capture images of documents and instructions used in a factory, digitize them, and manage them.

[1194] System Overview

[1195] An operator in a factory wears smart glasses and takes a photo of an instruction manual or instruction sheet. The captured image is processed by a dedicated application in the smart glasses and then uploaded to a server. The server extracts text information from the image, summarizes it, and registers it in a task management system. It also responds to the operator's questions and adjusts its response according to the operator's emotional state.

[1196] Hardware Configuration

[1197] Smart glasses: Worn by operators in the factory, they take photos of instructions and manuals.

[1198] Camera: Built into the smart glasses and equipped with a high-resolution camera.

[1199] Cloud server: Performs image processing, character recognition, summary generation, calendar registration, and sentiment analysis.

[1200] Task Management System: Functions as a calendar application, managing summary information and related date information.

[1201] Software Configuration

[1202] OCR software: Uses software such as Google Cloud Vision to extract text information from uploaded images.

[1203] Generative AI model: Summarize text information using OpenAI's GPT model.

[1204] Sentiment analysis engine: Using IBM Watson NLU and other technologies, it recognizes the emotional state of the user from their questions and messages and adjusts the response accordingly.

[1205] Dedicated application: Installed on smart glasses, it provides functions for taking photos, uploading images, and answering questions.

[1206] Processing flow

[1207] The server includes an uploading means for a user to take an image of a document and convert the image into data, a character recognition means for the server to extract character information from the uploaded image, a summary generation means for the server to summarize the extracted character information, a calendar registration means for the server to automatically register the summary information and related date information in a calendar application, a question and answer means for a user to ask a question to the system and the question is answered from the summary information or original data, an emotion analysis means for recognizing the emotional state of the user and adjusting the response, a means for taking an image of a document using a camera attached to the item processing device, and a transmission means for transmitting the taken image to the processing device.

[1208] Specific examples

[1209] For example, when an operator takes a photo of an instruction sheet using smart glasses, the image is first saved in a dedicated app on the smart glasses and then uploaded to a server. The server uses Google Cloud Vision for OCR processing to extract text information. The text is then summarized using OpenAI's GPT model, and the results are registered in the factory's task management system. When an operator asks, "What's the next task?", the Q&A system provides an answer based on the summary information and the original data. If the operator asks impatiently, "Why is it so slow?", IBM Watson NLU performs sentiment analysis and provides a polite response such as, "Sorry, we'll get back to you right away."

[1210] Prompt Sentence Examples

[1211] Summarize the following text:

[1212] The sports day will be held on July 15th at the school playground.

[1213] summary:

[1214] In this way, the invention makes it possible to improve the efficiency of document management within a factory and also to respond appropriately to the emotions of operators.

[1215] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1216] Process step description

[1217] (Step 1: User takes a photo of the document)

[1218] A user puts on the smart glasses and takes a photo of an instruction manual or instruction sheet. The input is the photographed paper document, and the output is a digital image file. Specifically, the glasses use a high-resolution camera to capture a clear image.

[1219] (Step 2: Save the image with the smart glasses)

[1220] The digital image file is stored in a dedicated application in the smart glasses. The input is the captured digital image, and the output is the image file stored in the smart glasses' storage. The specific operation is to perform the image file storage process.

[1221] (Step 3: Upload the image to the server)

[1222] The saved image files are uploaded to a server by a dedicated application in the smart glasses. The input is the image file saved in the smart glasses' storage, and the output is the image data sent to the server. Specifically, the data is sent via an internet connection.

[1223] (Step 4: Image processing and OCR on the server)

[1224] The server uses OCR software such as Google Cloud Vision to extract text information from uploaded images. The input is the image data uploaded to the server, and the output is text data with the text information. Specifically, OCR processing is performed to recognize the characters in the image.

[1225] (Step 5: Summary generation)

[1226] The server uses the character information extracted by OCR to generate a summary using OpenAI's GPT model. The input is text data extracted by OCR, and the output is a summarized text. The specific operation involves natural language processing using a generative AI model.

[1227] (Step 6: Register with your calendar application)

[1228] The server analyzes the summarized text information and related date information and automatically registers it in the factory's task management system. The input is the summarized text data, and the output is event information registered in a calendar application. Specific operations include text analysis and database registration.

[1229] (Step 7: Question and Answer System)

[1230] A user inputs a question into smart glasses or other devices, and the question is sent to a server. The server uses a question-and-answer system to generate an answer from summary information and the original data. The input is the user's question, and the output is the generated answer. Specifically, the system uses natural language processing technology to analyze the question and generate the answer.

[1231] (Step 8: Sentiment analysis and response adjustment)

[1232] The server analyzes the questions and messages entered by the user and uses an emotion analysis engine such as IBM Watson NLU to recognize the user's emotional state and adjust the response. The input is the user's question or message, and the output is a response that adjusts the emotional state and response. The specific operation involves emotion analysis and adjusting the response based on that.

[1233] This system not only improves the efficiency of document management within the factory, but also enables appropriate responses based on the emotions of operators.

[1234] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1235] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1236] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1237] [Fourth embodiment]

[1238] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1239] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1240] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1241] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1242] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1243] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1244] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1245] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1246] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1247] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1248] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1249] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1250] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1251] This invention is a system for efficiently managing a large amount of printed materials distributed by a school, and is specifically implemented as follows.

[1252] Overall system overview

[1253] The system automates a series of processes: users upload images of prints they have taken, extract text information from the images to generate summaries, register them in a calendar application, and then participate in a Q&A session.

[1254] Program processing

[1255] 1. Taking a photo and uploading it

[1256] The user takes a photo of the printouts their child brings home using their smartphone. After taking the photo, the user uploads the image to the server using a dedicated app for the system. By pressing the upload button, the image data is sent to the server.

[1257] 2. Image Processing and Character Recognition

[1258] The server temporarily stores the received image for processing. The server then uses OCR (optical character recognition) technology to recognize the characters in the image and convert them into text data. This process digitizes the contents of the paper printout.

[1259] 3. Summary of Contents

[1260] The server summarizes the text information extracted by OCR using a generative model (e.g., GPT). This summary generation method produces a short and concise summary of the long printed content.

[1261] 4. Register with your calendar app

[1262] The server analyzes the summarized information and automatically registers relevant dates and event information in a calendar application. For example, if the content says "The sports day will be held on the school playground on July 15th," the date "July 15th" will be identified and registered as an event called "Sports Day" in the calendar.

[1263] 5. Question and Answer System

[1264] The server stores the summary and the original text information in a database. When a user enters a question into the system using a smartphone or PC, the server searches the database for an answer to that question. For example, in response to a question such as "Where will the sports day be held?", the server will return an answer such as "The sports day will be held in the school's athletic field."

[1265] Specific examples

[1266] Example 1: Sports day announcement

[1267] A user takes a photo of a notice printout for a sports day and uploads it to the system. The server extracts the text information "The sports day will be held on July 15th at the school playground" from the image and summarizes it as "Sports day: July 15th" using a summary generation model. After that, an event called "Sports day" is registered in the calendar app for July 15th. When the user asks "What time does the sports day start?", the server replies "The sports day starts at 9:00 AM."

[1268] Example 2: School lunch menu

[1269] A user takes a photo of a school lunch menu and uploads it to the system. The server extracts text information from the image, such as "July 1st: curry rice, July 2nd: fish meunière," and summarizes it as "July menu" using a summary generation model. The calendar is registered with the school lunch menu for each date. When a user asks, "What's for lunch on July 2nd?", the server responds, "The school lunch on July 2nd is fish meunière."

[1270] This system allows users to quickly understand the contents of printed materials, making it easier to manage schedules. It also allows users to quickly get answers to their questions, significantly reducing the burden of organizing printed materials.

[1271] The processing flow will be explained below.

[1272] Step 1:

[1273] The user takes a photo of the print that the child brought home using a smartphone terminal.

[1274] Step 2:

[1275] The images taken by the terminal are imported into the "School Print Summary System" application.

[1276] Step 3:

[1277] The user presses the "Upload" button in the app to send the image data to the server.

[1278] Step 4:

[1279] The server receives the uploaded image data and temporarily stores it.

[1280] Step 5:

[1281] The server invokes the OCR engine to extract text information from the stored image data, specifically recognizing text areas within the image and converting them into text data.

[1282] Step 6:

[1283] The server uses a generative AI model (e.g., GPT) to summarize the extracted text information. Specifically, it extracts key points from long text data and generates a concise summary.

[1284] Step 7:

[1285] The server parses the summarized information to identify relevant date and event information, specifically identifying dates and event names within the text.

[1286] Step 8:

[1287] The server calls the calendar application's API and automatically registers the specified events and dates in the calendar. For example, using the Google Calendar API, it registers "Sports Day" on July 15th.

[1288] Step 9:

[1289] The server stores the summary information and the original text in a database, allowing for later question and answer sessions.

[1290] Step 10:

[1291] A user opens a chat window within the system and types a question, for example, "Where is the sports day being held?"

[1292] Step 11:

[1293] The server receives the user's question, searches for relevant information in the database, and generates the best answer based on the summary information and the original text information.

[1294] Step 12:

[1295] The server sends the generated answer to the user, for example, "The sports day will be held on the school grounds."

[1296] The system operates according to these steps, allowing users to efficiently manage print content and quickly obtain the information they need.

[1297] Example 1

[1298] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1299] Conventional document management systems lacked the means to efficiently manage the large number of printed materials distributed by schools, requiring users to manually check the contents and manage schedules. Furthermore, they were unable to effectively combine question and answer functions, making it difficult to quickly obtain information. This resulted in significant waste of time and effort.

[1300] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1301] In this invention, the server includes a prompt generation means for generating a prompt sentence based on character information acquired by OCR using a generative AI model, a summary generation means for sending the generated prompt sentence to the generative AI model and generating a summary, and a calendar registration means for automatically registering the summary information and related date information in a calendar application, thereby enabling a user to quickly understand the contents of a document, efficiently manage their schedule, and quickly obtain necessary information.

[1302] "User" refers to a person who uses the system to take images of documents and review and manage information.

[1303] "Documents" refers to physical documents such as handouts and notices distributed by the school.

[1304] "Image" refers to the digital photograph obtained when a user photographs a document.

[1305] "Server" refers to the central system that processes images uploaded by users and performs character recognition, summary generation, calendar registration, and question and answering.

[1306] "Uploading means" refers to a function or device for sending images taken by a user to a server.

[1307] "Character recognition means" refers to a function or device that extracts character information from uploaded images.

[1308] "Summary generation means" refers to a function or device that briefly summarizes extracted text information.

[1309] "Calendar registration means" refers to a function or device that automatically registers summarized information and related date information in a calendar application.

[1310] "Question and answering means" refers to a function or device that answers questions posed by users to the system using summary information or original data.

[1311] "OCR" stands for optical character recognition technology, which refers to the technology of extracting text information from images.

[1312] "Prompt generation means" refers to a function or device that generates a prompt sentence based on character information obtained by OCR.

[1313] A "generative AI model" refers to an artificial intelligence model that performs natural language processing and enables summarization and question-and-answering.

[1314] This invention is a system for efficiently managing large volumes of documents distributed by schools. This system automates a series of processes: users take photos of documents, upload them to a server, extract text information using OCR technology, generate summaries using a generative AI model, register the summaries in a calendar application, and then handle questions and answers.

[1315] Hardware and software used

[1316] Terminal

[1317] Users use their smartphones to take pictures of documents and then upload the images to the server using a dedicated application for the system.

[1318] server

[1319] The server implements the following functions:

[1320] 1. Character recognition means

[1321] The server uses OCR (Optical Character Recognition) technology, such as the Tesseract OCR library, to extract text information from the uploaded image.

[1322] 2. Prompt Generation Method

[1323] The server generates a prompt based on the text information obtained by OCR, which is then sent to a generative AI model to generate a summary.

[1324] 3. Summary generation means

[1325] The server summarizes text information using a generative AI model such as GPT-3. The server inputs the prompt sentence generated by the prompt generation means into the model and obtains the summary result.

[1326] 4. Calendar registration method

[1327] The server analyzes the summarized information to find relevant dates and events, and automatically registers them in a calendar application. For example, by using the Google Calendar API, extracted event information can be added to a calendar.

[1328] 5. Q&A tools

[1329] The server stores the summary information and the original text information in a database and provides appropriate answers to questions from users. When a user enters a question on a smartphone or PC, the server searches the database and returns the relevant information.

[1330] Specific examples

[1331] Example 1: Sports day announcement

[1332] A user takes a photo of a sports day announcement printout and uploads it to the system. The server extracts the text information "The sports day will be held on July 15th at the school playground" from the image and summarizes it as "Sports day: July 15th" using a generative AI model. Then, an event called "Sports day" is registered in a calendar app for July 15th. When the user asks, "What time does the sports day start?", the server responds with, "The sports day starts at 9:00 AM."

[1333] Example 2: School lunch menu

[1334] A user takes a photo of a school lunch menu and uploads it to the system. The server extracts text information from the image, such as "July 1st: Curry rice, July 2nd: Fish meunière," and uses a generative AI model to summarize it as "July menu." The calendar is then populated with school lunch menus for each date. When a user asks, "What's for lunch on July 2nd?", the server responds, "The school lunch on July 2nd is fish meunière."

[1335] Prompt Sentence Examples

[1336] An example prompt for a generative AI model to summarize is:

[1337] Example prompt 1:

[1338] "Please summarize the following sentence concisely: 'The sports day will be held on July 15th at the school playground. It will start at 9:00 AM. Participants should assemble in their respective classes.'"

[1339] Example prompt 2:

[1340] "Based on the following information, please generate a summary for calendar registration: 'July 1st: Curry rice, July 2nd: Fish meunière, July 3rd: Hamburg steak'"

[1341] This system allows users to quickly understand the contents of printed materials, making it easier to manage schedules. It also allows users to quickly get answers to their questions, significantly reducing the burden of organizing printed materials.

[1342] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1343] Step 1:

[1344] The user uses a smartphone to take a photo of the document that the child has brought home, opens the dedicated app, selects the image, and presses the upload button, which sends the image data to the server.

[1345] Input: Image data of a document photographed with a smartphone

[1346] Output: Image data sent to the server

[1347] Specific behavior:

[1348] The user launches the camera app on their smartphone and takes a photo of the document.

[1349] The user opens the system's dedicated app and selects the photos they have taken.

[1350] The user presses the upload button in the app to send the image data to the server.

[1351] Step 2:

[1352] When the server receives the uploaded image data, it temporarily stores it, then uses OCR (optical character recognition) technology to extract character information from the image and convert it into text data.

[1353] Input: Image data uploaded to the server

[1354] Output: Extracted character information (text data)

[1355] Specific behavior:

[1356] The server temporarily stores the image data in a defined directory.

[1357] The server runs an OCR engine (e.g., Tesseract) to extract text information from the image.

[1358] The extracted character information is saved as a text file and passed on to the next process.

[1359] Step 3:

[1360] The server generates a prompt sentence based on the character information acquired by OCR and the generative AI model, and sends this prompt sentence to the generative AI model to obtain a summary.

[1361] Input: Character information obtained by OCR

[1362] Output: Generated prompt statement and summary information

[1363] Specific behavior:

[1364] The server generates a prompt based on the character information obtained by OCR.

[1365] The server sends prompt sentences to the generative AI model and obtains summaries.

[1366] The generated summary information is passed to the next process.

[1367] Step 4:

[1368] The server analyzes the summarized information to find relevant dates and event information and automatically registers them in the calendar application.

[1369] Input: Summarized information (text data)

[1370] Output: Event information registered in the calendar

[1371] Specific behavior:

[1372] The server parses the dates and event names contained in the summary.

[1373] The server uses a calendar API (e.g., Google Calendar API) to add the event information to the calendar.

[1374] Step 5:

[1375] The server stores summary information and the original text information in a database. When a user enters a question on a smartphone or computer, the server searches the database and provides the appropriate answer.

[1376] Input: Question from user (text data)

[1377] Output: Answer to the question (text data)

[1378] Specific behavior:

[1379] The server stores the OCR and summary information in a database.

[1380] Users enter their questions on their smartphones or computers.

[1381] The server searches a database and returns an answer based on the question entered.

[1382] (Application example 1)

[1383] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1384] In brick-and-mortar stores, it is difficult to efficiently manage paper media such as shift schedules and promotion information used by store staff in their daily work. This results in problems such as time-consuming sharing and updating of information, which reduces work efficiency. There is also a lack of means to quickly search for necessary information and obtain answers. There is a need for a system that can solve these problems.

[1385] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1386] In this invention, the server includes an uploading means for a user to photograph information on paper media as an image and convert the image into data, a character recognition means for the server to extract text information from the uploaded image, a summary generation means for the server to summarize the extracted text information, a calendar registration means for the server to automatically register the summary information and related date information in a calendar application, a question and answer means for a user to ask the system questions and the system answers the questions using the summary information or original data, and a store data linking means for linking the extracted text information and summary to a store management system. This enables store staff to efficiently digitize information on paper media and quickly search and manage it.

[1387] "User" refers to a person who uses the system to digitize and manage paper-based information.

[1388] "Paper media" refers to physical documents or books on which text, charts, etc. are printed.

[1389] The "uploading means" refers to a function that allows a user to take an image of information on a paper medium and send the image data to a server.

[1390] "Character recognition means" refers to the function by which the server extracts character information from uploaded images, and typically uses optical character recognition technology.

[1391] "Summary generation means" refers to the function by which the server succinctly summarizes the extracted text information, and typically uses a generative AI model.

[1392] The "calendar registration means" refers to a function that automatically registers summary information and related date information in a calendar application.

[1393] "Question and answer means" refers to a function that allows users to ask questions to the system and receive answers from summary information or original data.

[1394] "Store data linking means" refers to the function of associating extracted text information and summaries with a store management system.

[1395] "Server" refers to a computer device that processes, stores, and performs corresponding functions for the entire system.

[1396] "Physical store" refers to a store that exists in a physical location and is managed and operated by store staff.

[1397] The present invention is a system for efficiently managing paper-based information in a physical store, and is specifically implemented as follows.

[1398] System Configuration

[1399] This system automates the process by which users digitize paper media in their stores (e.g., shift schedules, promotion information) and register that information in a store management system. Specifically, it consists of devices such as smartphones and tablets, a server, and various software.

[1400] Hardware and Software

[1401] Hardware:

[1402] Smartphone or tablet

[1403] server

[1404] software:

[1405] Image processing library (OpenCV)

[1406] Optical Character Recognition Library (Tesseract-OCR)

[1407] Generative AI model (HuggingFace Transformers T5 model)

[1408] Calendar event generation library (icalendar)

[1409] Operating procedure

[1410] 1. The user photographs a paper medium: The user uses a smartphone or tablet to take a picture of a paper medium such as a shift schedule or promotional information.

[1411] 2. Uploading image data: The captured image data is uploaded to the server via a dedicated application.

[1412] 3. Character recognition by the server: The server preprocesses the received image using an image processing library (OpenCV) and extracts character information using an optical character recognition library (Tesseract-OCR).

[1413] 4. Summary generation: The extracted text information is summarized using a generative AI model (e.g., the T5 model from HuggingFace Transformers).

[1414] 5. Calendar Registration: The summary information and associated date information are automatically registered in the calendar application using the calendar event generation library (icalendar).

[1415] 6. Linkage to store management system: The extracted text information and summaries are linked to the store management system and reflected in staff work shifts and promotion schedules.

[1416] 7. Question and Answer System: Users can input questions into the system and the server will provide answers to the questions from summary information and original data.

[1417] Specific examples

[1418] Example 1: Managing shift schedules

[1419] A user takes a photo of a shift schedule and uploads the image to the server. The server extracts text information from the image, such as "July 1st: Person A is on duty, July 2nd: Person B is on duty," and summarizes it as the "July shift schedule" using a generative AI model. Shift information for each date is registered on the calendar. When a user asks, "Who is on the July 2nd shift?" the server responds, "Person B is on the July 2nd shift."

[1420] Example prompt sentence:

[1421] What's the next promotion?

[1422] Example 2: Managing promotion information

[1423] A user takes a photo of a printed copy of promotional information and uploads the image to a server. The server extracts text information from the image, such as "10% off sale on January 10th," and summarizes it using a generative AI model. An event called "Promotion: 10% off sale (January 10th)" is then registered in a calendar app. When the user asks, "What's the next promotion?", the server responds, "The next promotion is a 10% off sale on January 10th."

[1424] This system allows staff at physical stores to efficiently digitize paper-based information, enabling smooth business management.

[1425] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1426] Step 1:

[1427] Users take a photo of paper-based information (such as shift schedules or promotional information) using a smartphone or tablet. This generates image data, which is then uploaded to a server via a dedicated application. The input is the captured image data, and the output is the uploaded image data.

[1428] Step 2:

[1429] The server receives the uploaded image data and temporarily stores it. This image data is prepared for the next processing. The input is the uploaded image data, and the output is the stored image data.

[1430] Step 3:

[1431] The server preprocesses the received image data using an image processing library (OpenCV). Specifically, it converts the image to grayscale and performs noise reduction. The input is the stored image data, and the output is the preprocessed image data.

[1432] Step 4:

[1433] The server extracts character information from the preprocessed image using an optical character recognition library (Tesseract-OCR). The input is the preprocessed image data, and the output is text data as character information.

[1434] Step 5:

[1435] The server uses a generative AI model (e.g., the T5 model from HuggingFace Transformers) to summarize the extracted text information. The input is text data as text information, and the output is summarized information.

[1436] Step 6:

[1437] The server uses a calendar event generation library (icalendar) to register the summary information and associated date information to the calendar application. The input is the summary information and date information, and the output is the event registered in the calendar.

[1438] Step 7:

[1439] The server associates the extracted text and summary information with the store management system, which updates shift information and promotion schedules. The input is text and summary information, and the output is updated store management data.

[1440] Step 8:

[1441] The user inputs a question to the system. The server searches for an appropriate answer from the summary information and the original data based on the question and provides it to the user. The input is the user's question, and the output is the answer to the question.

[1442] This allows users to efficiently digitize paper-based information, making it easier to manage information within the store.

[1443] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1444] This invention is a system for efficiently managing a large amount of printed materials distributed by schools, and further has a function for recognizing the user's emotions and adjusting the response accordingly. Specific embodiments of the invention are described below.

[1445] Overall system overview

[1446] This system automates a series of processes: uploading images of prints taken by users, extracting text information from the images to generate summaries, registering them in a calendar application, and answering questions. It also incorporates an emotion engine that recognizes emotions from questions and messages entered by users and adjusts responses accordingly.

[1447] Program processing

[1448] 1. Taking a photo and uploading it

[1449] The user takes a photo of the printouts their child brings home using their smartphone. After taking the photo, the user uploads the image to the server using a dedicated app for the system. By pressing the upload button, the image data is sent to the server.

[1450] 2. Image Processing and Character Recognition

[1451] The server temporarily stores the received image for processing. The server then uses OCR (optical character recognition) technology to recognize the characters in the image and convert them into text data. This process digitizes the contents of the paper printout.

[1452] 3. Summary of Contents

[1453] The server summarizes the text information extracted by OCR using a generative model (e.g., GPT). This summary generation method produces a short and concise summary of the long printed content.

[1454] 4. Register with your calendar app

[1455] The server analyzes the summarized information and identifies relevant dates and events. For example, if the content says "The sports day will be held on the school grounds on July 15th," the date "July 15th" is identified and an event called "Sports Day" is registered on the calendar.

[1456] 5. Question and Answer System

[1457] The server stores the summary and the original text information in a database. When a user enters a question into the system using a smartphone or PC, the server searches the database for an answer to that question. For example, in response to a question such as "Where will the sports day be held?", the server will return an answer such as "The sports day will be held in the school's athletic field."

[1458] 6. Leveraging Emotional Engines

[1459] The server analyzes the questions and messages entered by the user and uses an emotion engine to recognize the user's emotional state. For example, if the user enters "It's too late! Tell me quickly!", the emotion engine will determine that the user is irritated.

[1460] If the emotion engine recognizes the user's emotion, the server will adjust the content and tone of the response, for example, responding to an annoyed user with a polite and prompt "I'm sorry, let me check again."

[1461] Furthermore, a dashboard can be provided to visually display the emotional state recognized by the emotion engine, allowing users to monitor changes in their emotions in real time and respond appropriately.

[1462] Specific examples

[1463] Example 1: Sports day announcement

[1464] A user takes a photo of a sports day announcement printout and uploads it to the system. The server extracts the text information "The sports day will be held on July 15th at the school playground" from the image and summarizes it as "Sports day: July 15th" using a summary generation model. An event called "Sports day" is then registered in the calendar app for July 15th. When the user asks, "What time does the sports day start?", the server replies, "The sports day starts at 9:00 AM." If the user impatiently asks, "Why is it so late?", the server responds politely with, "Sorry, I'll check right away."

[1465] Example 2: School lunch menu

[1466] A user takes a photo of a school lunch menu and uploads it to the system. The server extracts text information from the image, such as "July 1st: Curry rice, July 2nd: Fish meunière," and summarizes it as "July menu" using a summary generation model. The calendar registers the school lunch menu for each date. If a user asks, "What's for lunch on July 2nd?", the server responds, "The school lunch on July 2nd is fish meunière." If the user complains that the server is slow to respond, the server responds, "Sorry for the wait, we'll get back to you right away."

[1467] In this way, by utilizing the emotion engine, this system can efficiently summarize, manage, and answer questions about information while responding appropriately to the user's emotions.

[1468] The processing flow will be explained below.

[1469] Step 1:

[1470] The user takes a photo of the print that the child brought home using a smartphone terminal.

[1471] Step 2:

[1472] The images taken by the terminal are imported into the "School Print Summary System" application.

[1473] Step 3:

[1474] The user presses the "Upload" button in the app to send the image data to the server.

[1475] Step 4:

[1476] The server receives the uploaded image data and temporarily stores it.

[1477] Step 5:

[1478] The server invokes the OCR engine to extract text information from the stored image data, specifically recognizing text areas within the image and converting them into text data.

[1479] Step 6:

[1480] The server uses a generative AI model (e.g., GPT) to summarize the extracted text information. Specifically, it extracts key points from long text data and generates a concise summary.

[1481] Step 7:

[1482] The server parses the summarized information to identify relevant date and event information, specifically identifying dates and event names within the text.

[1483] Step 8:

[1484] The server calls the calendar application's API and automatically registers the specified events and dates in the calendar. For example, using the Google Calendar API, it registers "Sports Day" on July 15th.

[1485] Step 9:

[1486] The server stores the summary information and the original text in a database, allowing for later question and answer sessions.

[1487] Step 10:

[1488] A user opens a chat window within the system and types a question, for example, "Where is the sports day being held?"

[1489] Step 11:

[1490] The server receives the user's question, searches for relevant information in the database, and generates the best answer based on the summary information and the original text information.

[1491] Step 12:

[1492] The server sends the generated answer to the user, for example, "The sports day will be held on the school grounds."

[1493] Step 13:

[1494] The server analyzes the user's questions and messages and uses an emotion engine to recognize the user's emotional state. For example, if the user types "You're late! Tell me quickly!", the server detects the emotion of irritation.

[1495] Step 14:

[1496] The server adjusts the content and tone of the response depending on the emotion recognized by the emotion engine. If it detects irritation, it will respond politely and promptly, saying something like, "I'm sorry, we'll look into it right away."

[1497] Step 15:

[1498] The server reflects the user's emotions recognized by the emotion engine on a visual dashboard, which is used by system administrators to monitor the user's emotional state in real time.

[1499] The system operates according to these steps, allowing users to efficiently manage print content and quickly obtain the information they need. It also provides a better user experience by responding according to the user's emotions.

[1500] Example 2

[1501] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1502] Conventional print management systems have difficulty efficiently digitizing large amounts of paper media and managing summaries and schedules. Responses to user questions are also limited to standardized content, and there are also issues with not being able to recognize user emotions and respond flexibly.

[1503] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1504] In this invention, the server includes uploading means for a user to take an image of a recording medium and digitize the image, character recognition means for the server to extract character information from the uploaded image, summary generation means for the server to summarize the extracted character information, calendar registration means for the server to automatically register the summary information and related date information in a time management application, question and answering means for the user to ask a question to the system and to answer the question from the summary information or original data, emotion recognition means for the server to analyze the question or message entered by the user and recognize the user's emotional state using an emotion engine, and tone adjustment means for adjusting the content and tone of the response based on the recognition result by the emotion recognition means. This enables efficient digitization, summarization, automatic schedule registration, and flexible response based on emotions.

[1505] The "uploading means" is a means by which a user takes an image of a recording medium and transmits the image as data to a server.

[1506] "Character Recognition Means" refers to the techniques or tools used by the server to extract text information from uploaded images.

[1507] "Summary generation" refers to a generative model or other technology that allows the server to summarize extracted textual information in a concise, short format.

[1508] The "calendar registration means" is a means by which the server automatically registers summary information and related date information in the time management application.

[1509] The "question and answer means" is a means by which a user can ask a question to the system and have the system provide an answer based on the summary information or the original data in response to the question.

[1510] "Emotion recognition means" refers to an engine or technology that allows the server to analyze questions and messages entered by the user and recognize the user's emotional state.

[1511] The "tone adjustment means" is a means by which the server responds to the user by adjusting the content and tone of the response based on the recognition result by the emotion recognition means.

[1512] The present invention is a system for efficiently managing a large number of printed materials distributed by schools, recognizing the emotions of users, and adjusting responses accordingly. Specific embodiments of this system are described below.

[1513] Hardware / Software used

[1514] 1. Smartphone device:

[1515] This is a device used by users to take photos of prints and upload the images to a server using a dedicated app.

[1516] 2. Server:

[1517] A computer system that performs a series of processes such as image processing, character recognition, summary generation, calendar entry, and emotion recognition. It also includes a database.

[1518] 3. OCR (Optical Character Recognition) technology:

[1519] This is a technology for extracting text information from images. Specifically, Tesseract OCR is used.

[1520] 4. Generative AI Models:

[1521] This is a generative model for summarizing text information. For example, GPT-4 is used.

[1522] 5. Calendar application:

[1523] It is a time management tool that registers summary information and date information. It uses APIs such as Google Calendar.

[1524] 6. Emotion Recognition Engine:

[1525] This is a technology for analyzing user emotions, and uses sentiment analysis APIs, etc.

[1526] Specific examples

[1527] Example 1: Sports day announcement

[1528] A user takes a photo of a sports day announcement printout and uploads it to the server using a dedicated app. The server performs OCR processing on the received image and extracts the text information, such as "The sports day will be held on July 15th at the school's athletic field." The server then uses a generative AI model to summarize the information as "Sports day: July 15th." The summary information is then registered on the calendar as an event called "Sports day" using the Google Calendar API.

[1529] When a user asks, "What time does the sports day start?", the server searches the database for relevant information and replies, "The sports day starts at 9 a.m." If the user asks impatiently, "Why is it so late?", the emotion recognition engine will recognize the impatience and the server will respond flexibly, saying, "I'm sorry, I'll check right away."

[1530] Example 2: School lunch menu

[1531] The user takes a photo of the school lunch menu and uploads it to the server using a dedicated app. The server performs OCR on the image and extracts the text information "July 1st: Curry rice, July 2nd: Fish meunière." A generative AI model is used to summarize it as "July menu." The school lunch menu for each date is then registered on the calendar.

[1532] When a user asks, "What's for lunch on July 2nd?", the server responds, "The school lunch on July 2nd is fish meunière." If the user expresses dissatisfaction by saying, "You're taking so long to respond!", the emotion recognition engine recognizes the dissatisfaction, and the server responds, "Sorry to keep you waiting, we'll get back to you right away."

[1533] Prompt Sentence Examples

[1534] Example prompt 1:

[1535] "This is a printout announcing the sports day. Please summarize the following information and add it to your calendar. Content: 'The sports day will be held on July 15th at the school playground.'"

[1536] Example prompt 2:

[1537] "I have the school lunch menu in front of me. Please register the following school lunch menu items on your calendar. Contents: July 1st: Curry rice, July 2nd: Fish meunière."

[1538] As described above, this system efficiently digitizes, summarizes, and manages schedules based on user-taken print images. It also recognizes user emotions and responds flexibly accordingly. In this way, advanced OCR technology, generative AI models, and emotion recognition engines work together to improve the effectiveness of the entire system.

[1539] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1540] Step 1:

[1541] Taking photos and uploading

[1542] A user uses a smartphone device to take a photo of a printout that a child has brought home, for example, a printout listing a school event schedule.

[1543] Input: Printed image on paper

[1544] Output: Digital image files (JPEG, PNG, etc.)

[1545] Specific operation: The user launches the camera app on their smartphone, adjusts the screen so that the entire print fits on the screen, and takes a picture. After taking the picture, the image file is saved on the device.

[1546] Step 2:

[1547] Upload

[1548] The user launches the system's dedicated app and uploads the captured image to the server. Specifically, the user taps the "upload" button in the app, and the image is sent to the server.

[1549] Input: Digital image file

[1550] Output: Temporary file on the server

[1551] Specific operation: When the user taps the "Upload" button in the app, the selected image is sent to the server, where it is temporarily stored.

[1552] Step 3:

[1553] Image processing and character recognition

[1554] The server processes the uploaded image using OCR (Optical Character Recognition) technology to extract the text information from the image. Specifically, it uses tools such as Tesseract OCR to generate text data from the image.

[1555] Input: Temporary file on the server (digital image)

[1556] Output: Extracted character information (text data)

[1557] Specific operation: The server reads the saved image file and applies OCR technology to extract text information. The extracted text data is stored on the server.

[1558] Step 4:

[1559] Summary of contents

[1560] The server summarizes the text information extracted by OCR using a generative model (e.g., GPT-4). It provides a prompt such as "Please summarize the following content:"

[1561] Input: Extracted text information

[1562] Output: Summarized text information

[1563] Specific operation: The server inputs the extracted text information into the generative AI model and processes it along with the prompt "Please summarize the following content:" The generated summary is stored on the server.

[1564] Step 5:

[1565] Registering with a calendar app

[1566] The server identifies events and date information from the summarized information and automatically registers them in the calendar using an API such as Google Calendar.

[1567] Input: Summarized text information

[1568] Output: Event information registered in the calendar app

[1569] Specific operation: The server extracts the date, time, and event name from the summary text and registers the identified information in the calendar via the Google Calendar API. For example, "Sports Day: July 15th" is added to the calendar.

[1570] Step 6:

[1571] Q&A System

[1572] Users enter questions into the system using their smartphones or computers, and the server searches a database for answers to those questions and provides them.

[1573] Input: User question (text)

[1574] Output: Answer from the server (text)

[1575] Specific operation: When a user enters a question text through an app or web interface, the server searches the database for relevant information, generates an appropriate answer, and returns it to the user. For example, in response to the question, "What time does the sports day start?", the server responds with, "The sports day starts at 9:00 AM."

[1576] Step 7:

[1577] Utilizing the Emotion Engine

[1578] The server analyzes the questions and messages entered by the user, uses an emotion engine to recognize the user's emotional state, and adjusts the content and tone of the response based on the emotion recognition results.

[1579] Input: User question or message (text)

[1580] Output: Emotion recognition results and tailored responses (text)

[1581] What it does: The server extracts emotions from the user's text through an emotion analysis API, and if it detects irritation or frustration, it responds with a tailored response such as "Sorry, we'll take a look at it right away."

[1582] Emotional states are displayed in real time on a dashboard.

[1583] (Application example 2)

[1584] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1585] Conventional document management systems have difficulty efficiently digitizing and managing large volumes of documents, making the management of factory instructions and manuals particularly cumbersome. Furthermore, they do not respond to the emotions of operators, which can lead to problems such as reduced production efficiency and increased likelihood of work errors. The objective of this invention is to solve these problems by providing a system that responds appropriately to emotions while improving operator efficiency.

[1586] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: uploading means for a user to photograph an image of a document and digitize the image; character recognition means for the server to extract text information from the uploaded image; summary generation means for the server to summarize the extracted text information; calendar registration means for the server to automatically register the summary information and related date information in a calendar application; question and answer means for a user to ask the system questions and the system to answer the questions using the summary information or the original data; emotion analysis means for recognizing the user's emotional state and adjusting the response; means for photographing an image of a document using a photographing device attached to the item processing device; and transmission means for transmitting the photographed image to the processing device. This not only improves the efficiency of document management in the factory but also enables appropriate responses according to the operator's emotions.

[1587] The "uploading means" is a means for converting images taken by a user into data and transmitting the data to a server.

[1588] "Character recognition means" refers to a technology that extracts character information from uploaded images, and refers to optical character recognition technology.

[1589] The "summary generation means" is a technique for concisely summarizing extracted text information using a generative model.

[1590] The "calendar registration means" is a technique for automatically registering summary information and related date information in a calendar application.

[1591] "Question and answering means" is a technology that provides answers to questions posed by users to the system from summary information or original data.

[1592] "Emotion analysis means" is a technology that recognizes the emotional state of a user from questions or messages entered by the user and adjusts the response accordingly.

[1593] "Image capture device" refers to a device attached to an article processing device that captures an image of a document.

[1594] The "transmission means" is a means for transmitting the captured image to the processing device.

[1595] "Image processing means" refers to a series of technologies that analyze uploaded images and convert them into data.

[1596] The "information management system" is a series of systems that upload users' images, extract information from those images, summarize it, register it on a calendar, answer users' questions, and analyze users' emotions to adjust the response.

[1597] The following describes an embodiment of the present invention: This system uses smart glasses to capture images of documents and instructions used in a factory, digitize them, and manage them.

[1598] System Overview

[1599] An operator in a factory wears smart glasses and takes a photo of an instruction manual or instruction sheet. The captured image is processed by a dedicated application in the smart glasses and then uploaded to a server. The server extracts text information from the image, summarizes it, and registers it in a task management system. It also responds to the operator's questions and adjusts its response according to the operator's emotional state.

[1600] Hardware Configuration

[1601] Smart glasses: Worn by operators in the factory, they take photos of instructions and manuals.

[1602] Camera: Built into the smart glasses and equipped with a high-resolution camera.

[1603] Cloud server: Performs image processing, character recognition, summary generation, calendar registration, and sentiment analysis.

[1604] Task Management System: Functions as a calendar application, managing summary information and related date information.

[1605] Software Configuration

[1606] OCR software: Uses software such as Google Cloud Vision to extract text information from uploaded images.

[1607] Generative AI model: Summarize text information using OpenAI's GPT model.

[1608] Sentiment analysis engine: Using IBM Watson NLU and other technologies, it recognizes the emotional state of the user from their questions and messages and adjusts the response accordingly.

[1609] Dedicated application: Installed on smart glasses, it provides functions for taking photos, uploading images, and answering questions.

[1610] Processing flow

[1611] The server includes an uploading means for a user to take an image of a document and convert the image into data, a character recognition means for the server to extract character information from the uploaded image, a summary generation means for the server to summarize the extracted character information, a calendar registration means for the server to automatically register the summary information and related date information in a calendar application, a question and answer means for a user to ask a question to the system and the question is answered from the summary information or original data, an emotion analysis means for recognizing the emotional state of the user and adjusting the response, a means for taking an image of a document using a camera attached to the item processing device, and a transmission means for transmitting the taken image to the processing device.

[1612] Specific examples

[1613] For example, when an operator takes a photo of an instruction sheet using smart glasses, the image is first saved in a dedicated app on the smart glasses and then uploaded to a server. The server uses Google Cloud Vision for OCR processing to extract text information. The text is then summarized using OpenAI's GPT model, and the results are registered in the factory's task management system. When an operator asks, "What's the next task?", the Q&A system provides an answer based on the summary information and the original data. If the operator asks impatiently, "Why is it so slow?", IBM Watson NLU performs sentiment analysis and provides a polite response such as, "Sorry, we'll get back to you right away."

[1614] Prompt Sentence Examples

[1615] Summarize the following text:

[1616] The sports day will be held on July 15th at the school playground.

[1617] summary:

[1618] In this way, the invention makes it possible to improve the efficiency of document management within a factory and also to respond appropriately to the emotions of operators.

[1619] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1620] Process step description

[1621] (Step 1: User takes a photo of the document)

[1622] A user puts on the smart glasses and takes a photo of an instruction manual or instruction sheet. The input is the photographed paper document, and the output is a digital image file. Specifically, the glasses use a high-resolution camera to capture a clear image.

[1623] (Step 2: Save the image with the smart glasses)

[1624] The digital image file is stored in a dedicated application in the smart glasses. The input is the captured digital image, and the output is the image file stored in the smart glasses' storage. The specific operation is to perform the image file storage process.

[1625] (Step 3: Upload the image to the server)

[1626] The saved image files are uploaded to a server by a dedicated application in the smart glasses. The input is the image file saved in the smart glasses' storage, and the output is the image data sent to the server. Specifically, the data is sent via an internet connection.

[1627] (Step 4: Image processing and OCR on the server)

[1628] The server uses OCR software such as Google Cloud Vision to extract text information from uploaded images. The input is the image data uploaded to the server, and the output is text data with the text information. Specifically, OCR processing is performed to recognize the characters in the image.

[1629] (Step 5: Summary generation)

[1630] The server uses the character information extracted by OCR to generate a summary using OpenAI's GPT model. The input is text data extracted by OCR, and the output is a summarized text. The specific operation involves natural language processing using a generative AI model.

[1631] (Step 6: Register with your calendar application)

[1632] The server analyzes the summarized text information and related date information and automatically registers it in the factory's task management system. The input is the summarized text data, and the output is event information registered in a calendar application. Specific operations include text analysis and database registration.

[1633] (Step 7: Question and Answer System)

[1634] A user inputs a question into smart glasses or other devices, and the question is sent to a server. The server uses a question-and-answer system to generate an answer from summary information and the original data. The input is the user's question, and the output is the generated answer. Specifically, the system uses natural language processing technology to analyze the question and generate the answer.

[1635] (Step 8: Sentiment analysis and response adjustment)

[1636] The server analyzes the questions and messages entered by the user and uses an emotion analysis engine such as IBM Watson NLU to recognize the user's emotional state and adjust the response. The input is the user's question or message, and the output is a response that adjusts the emotional state and response. The specific operation involves emotion analysis and adjusting the response based on that.

[1637] This system not only improves the efficiency of document management within the factory, but also enables appropriate responses based on the emotions of operators.

[1638] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1639] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1640] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1641] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1642] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1643] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1644] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1645] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1646] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1647] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1648] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1649] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1650] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1651] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1652] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1653] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1654] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1655] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1656] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1657] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1658] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1659] The following is further disclosed regarding the above embodiment.

[1660] (Claim 1)

[1661] An uploading means for a user to take an image of a document and convert the image into data;

[1662] character recognition means for extracting character information from the uploaded image by the server;

[1663] a summary generating means for summarizing the extracted character information by the server;

[1664] a calendar registration means for automatically registering the summary information and the associated date information in a calendar application by the server;

[1665] A question and answering means for allowing a user to ask a question to the system and for the system to provide answers to the question from summary information or original data;

[1666] A system including:

[1667] (Claim 2)

[1668] 10. The system of claim 1, wherein the character recognition means is based on optical character recognition technology.

[1669] (Claim 3)

[1670] 2. The system of claim 1, wherein the summary generating means summarizes the content based on a generative model.

[1671] "Example 1"

[1672] (Claim 1)

[1673] An uploading means for a user to take an image of a document and convert the image into data;

[1674] character recognition means for extracting character information from the uploaded image by the server;

[1675] a summary generating means for summarizing the extracted character information by the server;

[1676] a calendar registration means for automatically registering the summary information and the associated date information in a calendar application by the server;

[1677] A question and answering means for allowing a user to ask a question to the system and for the system to provide answers to the question from summary information or original data;

[1678] a prompt generation means for generating a prompt sentence based on the character information acquired by the OCR and a generation AI model;

[1679] a summary generation means for transmitting the generated prompt sentence to the generative AI model by the server and generating a summary;

[1680] A system including:

[1681] (Claim 2)

[1682] 10. The system of claim 1, wherein the character recognition means is based on optical character recognition technology.

[1683] (Claim 3)

[1684] 2. The system of claim 1, wherein the summary generating means summarizes the content based on a generative model.

[1685] "Application Example 1"

[1686] (Claim 1)

[1687] An uploading means for a user to take an image of information on a paper medium and convert the image into data;

[1688] character recognition means for extracting character information from the uploaded image by the server;

[1689] a summary generating means for summarizing the extracted character information by the server;

[1690] a calendar registration means for automatically registering the summary information and the associated date information in a calendar application by the server;

[1691] A question and answering means for allowing a user to ask a question to the system and for the system to provide answers to the question from summary information or original data;

[1692] a store data linking means for linking the extracted character information and summary with a store management system;

[1693] A system including:

[1694] (Claim 2)

[1695] 10. The system of claim 1, wherein the character recognition means is based on optical character recognition technology.

[1696] (Claim 3)

[1697] 10. The system of claim 1, wherein the summary generating means summarizes the content based on a generative AI model.

[1698] "Example 2: Combining Emotion Engines"

[1699] (Claim 1)

[1700] an uploading means for allowing a user to take an image of a recording medium and convert the image into data;

[1701] character recognition means for extracting character information from the uploaded image by the server;

[1702] a summary generating means for summarizing the extracted character information by the server;

[1703] a calendar registration means for automatically registering the summary information and related date information in the time management application by the server;

[1704] A question and answering means for allowing a user to ask a question to the system and for the system to provide answers to the question from summary information or original data;

[1705] an emotion recognition means for the server to analyze questions and messages input by the user and recognize the emotional state of the user using an emotion engine;

[1706] a tone adjustment means for adjusting the content and tone of a response based on the recognition result by the emotion recognition means;

[1707] A system including:

[1708] (Claim 2)

[1709] 10. The system of claim 1, wherein the character recognition means is based on optical character recognition technology.

[1710] (Claim 3)

[1711] 2. The system of claim 1, wherein the summary generating means summarizes the content based on a generative model.

[1712] "Application example 2 when combining emotion engines"

[1713] (Claim 1)

[1714] An uploading means for a user to take an image of a document and convert the image into data;

[1715] character recognition means for extracting character information from the uploaded image by the server;

[1716] a summary generating means for summarizing the extracted character information by the server;

[1717] a calendar registration means for automatically registering the summary information and the associated date information in a calendar application by the server;

[1718] A question and answering means for allowing a user to ask a question to the system and for the system to provide answers to the question from summary information or original data;

[1719] emotion analysis means for recognizing the user's emotional state and adjusting the response;

[1720] means for capturing an image of the document using a capture device attached to the article processing device;

[1721] a transmitting means for transmitting the captured image to the processing device;

[1722] A system including:

[1723] (Claim 2)

[1724] 10. The system of claim 1, wherein the character recognition means is based on optical character recognition technology.

[1725] (Claim 3)

[1726] 2. The system of claim 1, wherein the summary generating means summarizes the content based on a generative model. [Explanation of symbols]

[1727] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. An uploading means for a user to take an image of a document and convert the image into data; character recognition means for extracting character information from the uploaded image by the server; a summary generating means for summarizing the extracted character information by the server; a calendar registration means for automatically registering the summary information and the associated date information in a calendar application by the server; A question and answering means for allowing a user to ask a question to the system and for the system to provide answers to the question from summary information or original data; A system including:

2. 2. The system of claim 1, wherein the character recognition means is based on optical character recognition technology.

3. 2. The system of claim 1, wherein the summary generating means summarizes the content based on a generative model.

Images