System

Abstract

A system is provided.SOLUTION: A system comprising: means for extracting character information from an image acquired from a camera by a character recognition engine; means for analyzing the extracted character information and surrounding background information to identify a suggested task to be performed next; means for notifying a user of the suggested task; and means for executing the suggested task based on user input.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Existing OCR (optical character recognition) applications require users to use different dedicated applications for different tasks, resulting in poor usability and cumbersome operation. Furthermore, they lack an auto-suggestion function to efficiently handle multiple tasks in a single application, forcing users to manually determine which task to perform next. This invention aims to solve these problems and improve user convenience. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by the following means. The system includes a means for extracting text information from an image acquired by a camera using a character recognition engine, and then a means for analyzing the extracted text information and surrounding background information to identify a suggested next task to be performed. The system further includes a means for notifying a user of the suggested task and a means for executing the suggested task based on the user's input. The system also includes a means for using multi-modality artificial intelligence to analyze the surrounding background information, and a means for recording the results of the suggested task and using the results as learning data to improve the accuracy of the next suggestion. This allows users to efficiently perform a variety of tasks using a single application.

[0006] A "character recognition engine" is a software or hardware configuration that has the function of extracting character information from image data acquired by a camera.

[0007] An "image" is digital data containing visual information captured by a camera.

[0008] "Text information" refers to the text data contained in the image, and is a collection of characters extracted by the OCR engine.

[0009] "Background information" refers to elements other than text information in an image captured by a camera, such as the visual context including color, shape, and layout.

[0010] A "suggested task" is a task that identifies the next action the user should take based on text and contextual information and is presented to the user.

[0011] "Notification" refers to a display method used to notify a user of a proposed task, and may include forms such as a pop-up window, push notification, or alert.

[0012] "User input" means a choice or decision made by a user in response to a proposed task, including, for example, tapping, clicking, or other actions.

[0013] The "execution means" refers to a function for actually processing a task selected by the user, and includes processes such as launching a corresponding external application and transferring data.

[0014] "Multi-modality artificial intelligence" is a multi-functional AI technology for the integrated analysis of text information and background information.

[0015] "Learning data" is data that records the results after performing a proposed task and serves as the basis for improving the accuracy of the next proposal.

[0016] "System" means a set of mechanical or electronic devices that operate in coordination with the above components. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9]1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] An embodiment of the present invention will be described below. The present invention provides an OCR application that uses a camera to recognize characters, analyzes surrounding information, and suggests and executes the next task to be performed.

[0039] 1. Camera image acquisition and preprocessing

[0040] The device activates the camera and captures the image taken by the user. For example, the user activates a scanner application and takes a picture of a receipt or document with the camera. The image is temporarily stored on the device and then sent to the server. The server preprocesses the received image, removing noise and optimizing the resolution to improve the accuracy of character recognition.

[0041] Examples:

[0042] The user takes a photo of the receipt, and the image is sent from the device to a server where noise is removed and resolution is adjusted.

[0043] 2. Character Recognition Processing

[0044] The server inputs the preprocessed image data into a character recognition engine. The OCR engine (e.g., an open-source OCR tool or a cloud-based OCR API) recognizes the characters in the image and extracts them as text data. The extracted text is then checked for spelling and grammar before being formatted into accurate data.

[0045] Examples:

[0046] The server extracts text information such as the product name, price, and store name from the receipt image.

[0047] 3. Analysis of surrounding information and task proposal

[0048] The server inputs the text information extracted by the OCR engine and the surrounding background information into a multi-modality AI. The AI ​​analyzes this information and identifies the most relevant tasks for the user. As a result of the analysis, a task list is generated and sorted by priority. This task list is then converted into a format that can be displayed to the user and sent to the device.

[0049] Examples:

[0050] The server uses the extracted receipt information and background information to suggest tasks such as "register this expenditure in a household accounting app" or "save a coupon."

[0051] 4. User Notification and Task Execution

[0052] The device notifies the user of the task list sent from the server. The notification is displayed in the form of a pop-up window or an alert. When the user selects one of the suggested tasks, the device executes that task. For example, it may register data in a household accounting app or launch an email sending app.

[0053] Examples:

[0054] When the user selects "Register in household accounting app," the device opens the household accounting app and enters the receipt information.

[0055] 5. Feedback of results

[0056] Once the task is completed, the device sends the result (success, failure, feedback) to the server, which stores the received feedback in a database and uses it as training data for the AI ​​model to improve the accuracy of the next task suggestions.

[0057] Examples:

[0058] If the data is successfully registered in the household accounting app, the information is sent to the server and used for analyzing receipt information the next time.

[0059] This embodiment allows users to efficiently handle various tasks with one OCR application, eliminating the need to use multiple dedicated applications. Furthermore, the use of multi-modality artificial intelligence improves the accuracy of suggested tasks and makes user operation easier and more intuitive.

[0060] The processing flow will be explained below.

[0061] Step 1:

[0062] The user launches the OCR application and switches to camera mode. The user presses the capture button to capture an image.

[0063] Step 2:

[0064] The captured image is temporarily stored on the device and then sent to the server.

[0065] Step 3:

[0066] The server decodes the received image into a file format and performs preprocessing such as adjusting the resolution and removing noise.

[0067] Step 4:

[0068] The server inputs the preprocessed image data into a character recognition engine, and the OCR engine extracts character information from the image.

[0069] Step 5:

[0070] The server performs spelling and grammar checks on the extracted text information and formats it into accurate text data.

[0071] Step 6:

[0072] The server inputs the text information and surrounding background information into the multi-modality artificial intelligence, and the AI ​​begins its analysis.

[0073] Step 7:

[0074] Based on the analysis results, the server's AI generates suggested tasks that are most relevant to the user and creates a task list sorted by priority.

[0075] Step 8:

[0076] The server formats the generated task list and sends it to the terminal for notification to the user.

[0077] Step 9:

[0078] The device displays the task list to the user as a popup window or alert, and the user selects one of the suggested tasks.

[0079] Step 10:

[0080] The device executes the corresponding task based on the user's selection, for example, launching a household accounting app and entering text data.

[0081] Step 11:

[0082] The server receives the task execution results (success, failure, feedback) from the terminal and stores them in a database.

[0083] Step 12:

[0084] The server uses the saved result data as learning data for the AI ​​and updates the model to improve the accuracy of the next proposed task.

[0085] Example 1

[0086] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0087] Conventional OCR applications have insufficient character recognition accuracy, often requiring users to manually correct information. Furthermore, information analysis and task suggestions after character recognition are ineffective, resulting in poor user convenience. Furthermore, the accuracy of the suggested tasks is low, and there is a lack of a learning mechanism that utilizes feedback.

[0088] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0089] In this invention, the server includes means for temporarily saving images acquired from the camera in an internal storage device, means for transmitting images from the internal storage device to the server and performing noise reduction and resolution optimization, means for using a character recognition engine to extract character information from the noise-removed and resolution-optimized images, means for analyzing the extracted character information and surrounding background information using multi-modality artificial intelligence and identifying a suggested next task to be performed, means for notifying the user of the suggested task, means for executing the suggested task based on user input, and means for sending feedback after task execution to the server and storing the feedback in a database to use as training data. This enables improved character recognition accuracy, effective information analysis and task suggestions, and a learning mechanism using feedback.

[0090] A "camera" is a device for taking still images and videos.

[0091] "Internal storage" means a storage medium within a device that is used to store data temporarily or permanently.

[0092] A "server" is a computer system that provides services to other computers on a network.

[0093] "Noise reduction" is the process of removing unwanted interference and noise from image and audio data.

[0094] "Resolution optimization" is a process of adjusting the image so that the details can be displayed clearly.

[0095] A "character recognition engine" is software or an algorithm that identifies characters in an image and extracts them as text data.

[0096] "Multi-modality artificial intelligence" is artificial intelligence that integrates, understands, and analyzes data in multiple different formats (e.g., text and images).

[0097] "Analysis" is the process of examining data in detail to understand its meaning and structure.

[0098] A "suggested task" is a specific action or task that is shown to the user to take next.

[0099] "Notification means" refers to a method or device for notifying a user of information.

[0100] "Feedback" refers to reporting the results of a task and user evaluation information to the server, and is used for future improvements.

[0101] A "database" is a system for efficiently storing, managing, and searching data.

[0102] An embodiment of the present invention will be described in detail. The invention provides an OCR application that uses a camera to recognize characters, analyzes surrounding information, and proposes and executes the next task to be performed. The following describes the flow of data processing and data calculation using each hardware and software.

[0103] 1. Camera image acquisition and preprocessing

[0104] The device activates the camera and acquires the image taken by the user. For example, the user activates a scanner application and takes a picture of a receipt or document with the camera. This image is temporarily stored in the device's internal storage and then sent to the server. The server preprocesses the received image, removing noise and optimizing the resolution to improve the accuracy of character recognition. For example, it uses the image processing library "OpenCV" to remove noise and optimizes the resolution using the "PhotoShop API."

[0105] Examples:

[0106] The user takes a photo of the receipt, and the image is sent from the device to a server where noise is removed and resolution is adjusted.

[0107] 2. Character Recognition Processing

[0108] The server inputs the preprocessed image data into a character recognition engine. The OCR engine used can be the open source "Tesseract" or the cloud-based "Google Cloud Vision API." The server calls the OCR engine, recognizes the characters in the image, and extracts them as text data. The extracted text is then checked for spelling and grammar using a natural language processing tool (such as "NLTK" or "SpaCy"), and formatted into accurate text data.

[0109] Examples:

[0110] The server extracts text information such as the product name, price, and store name from the receipt image.

[0111] 3. Analysis of surrounding information and task proposal

[0112] The server inputs the text information extracted by the OCR engine and the surrounding background information into a multi-modality AI (such as GPT-3 or BERT). The AI ​​analyzes this information and identifies the most relevant tasks for the user. As a result of the analysis, a task list is generated and sorted by priority. This task list is then converted into a format that can be displayed to the user and sent to the device.

[0113] Examples:

[0114] The server uses the extracted receipt information and background information to suggest tasks such as "register this expenditure in a household accounting app" or "save a coupon."

[0115] 4. User Notification and Task Execution

[0116] The device notifies the user of the task list sent from the server. The notification is displayed in the form of a pop-up window or an alert. When the user selects one of the suggested tasks, the device executes that task. Specifically, it registers data in a household accounting app or launches an email sending app.

[0117] Examples:

[0118] When the user selects "Register in household accounting app," the device opens the household accounting app and inputs the receipt information.

[0119] 5. Feedback of results

[0120] Once the task is completed, the device sends the result (success, failure, feedback) to the server, which stores the received feedback in a database and uses it as training data for the AI ​​model to improve the accuracy of the next task suggestions.

[0121] Examples:

[0122] If the data is successfully registered in the household accounting app, the information is sent to the server and used for analyzing receipt information the next time.

[0123] Prompt Sentence Examples

[0124] In a system for implementing the invention, more specific task suggestions can be made by using prompt sentences such as the following to input to a generative AI model:

[0125] example:

[0126] "I took a photo of my receipt. Can you give me some suggestions to help me manage my household finances and coupons?"

[0127] The above is an embodiment of the present invention. This embodiment allows users to efficiently handle a variety of tasks with a single OCR application, eliminating the need to use multiple dedicated applications. Furthermore, by utilizing multi-modality artificial intelligence, the accuracy of the suggested tasks is improved, and user operation becomes simpler and more intuitive.

[0128] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0129] Step 1:

[0130] A user starts a scanner application on their device and uses the camera to take a picture of a receipt or document. At this time, the device's camera captures the image data and temporarily stores it in the device's internal storage. The input is the receipt image taken by the user, and the output is the image data stored in the internal storage.

[0131] Step 2:

[0132] The device sends image data stored in its internal storage device to the server. The image data is sent over the Internet using a secure communication protocol such as HTTPS. The input is the image data in the internal storage device, and the output is the image data sent to the server.

[0133] Step 3:

[0134] The server analyzes the image data it receives and performs preprocessing to remove noise and optimize resolution. Specifically, it removes noise using the image processing library "OpenCV" and optimizes resolution using "PhotoShop API." The input is the image data sent to the server, and the output is image data with noise removed and resolution optimized.

[0135] Step 4:

[0136] The server inputs the preprocessed image data into a character recognition engine. The OCR engine used is "Tesseract" or "Google Cloud Vision API." The server uses the OCR engine to recognize characters in the image and extract them as text data. The input is noise-removed and resolution-optimized image data, and the output is the extracted text data.

[0137] Step 5:

[0138] The server uses natural language processing tools (such as NLTK and SpaCy) to check the spelling and grammar of the extracted text data, which then converts it into accurate text data. The input is text data obtained from the OCR engine, and the output is accurate text data that has been checked for spelling and grammar.

[0139] Step 6:

[0140] The server inputs the text data extracted by the OCR engine and the surrounding context into a multi-modality AI model (e.g., GPT-3 or BERT). The AI ​​model analyzes this information and identifies tasks that are highly relevant to the user. The input is the formatted text data and context, and the output is a list of suggested tasks.

[0141] Step 7:

[0142] The server converts the generated task list into a notification format and sends it to the terminal. For example, it constructs the task list in JSON format and sends it to the terminal via the Internet. The input is the specified task list, and the output is the task list sent to the terminal.

[0143] Step 8:

[0144] The terminal analyzes the task list received from the server and notifies the user. The notification is displayed in the form of a pop-up window or an alert. The input is the task list sent from the server, and the output is the tasks to be notified to the user.

[0145] Step 9:

[0146] The user selects one of the proposed tasks, for example, registering data in a household accounting app. The input is the notified task list, and the output is the specific task selected.

[0147] Step 10:

[0148] The device executes the selected task. Specifically, it opens a household accounting app and automatically inputs the receipt information the user wants to register. The input is the selected task information, and the output is the executed task.

[0149] Step 11:

[0150] The terminal sends the result (success, failure, feedback) after completing the task to the server. The result data is sent to the server via the Internet. The input is the result of the task execution, and the output is the feedback data sent to the server.

[0151] Step 12:

[0152] The server stores the received feedback in a database. The feedback data is used as training data for the AI ​​model to improve the accuracy of the next task suggestion. The input is the feedback data sent from the device, and the output is the updated database information.

[0153] (Application example 1)

[0154] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0155] In physical stores, there is a need for support to enable users to quickly and accurately obtain product information and choose appropriate actions based on that information. However, current systems require manual information collection and analysis, which is extremely time-consuming. In addition, there is a lack of systems that allow users to easily compare prices and product reviews at other stores while shopping in a store. Therefore, technology is needed to enable users to shop efficiently and effectively.

[0156] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0157] In this invention, the server includes means for extracting text information from an image acquired by a camera using a character recognition engine, means for analyzing the extracted text information and surrounding background information and identifying a suggested next task to be performed, means for notifying the user of the suggested task, means for executing the suggested task based on the user's input, means for performing preprocessing such as image noise reduction and resolution optimization, means for performing cloud-based analysis, and means for suggesting tasks such as comparing product prices and displaying product reviews. This enables users to make optimal shopping decisions in real time based on product and shelf information captured by the camera.

[0158] A "character recognition engine" is software or hardware that recognizes characters from images acquired from a camera or other imaging device and extracts them as digital text data.

[0159] "Preprocessing" refers to processing of the captured image, such as noise removal and resolution optimization, to improve the accuracy of character recognition.

[0160] "Cloud-based analysis" is a technology that uses computing resources in a cloud environment to analyze and calculate data, enabling high-speed, large-scale data processing.

[0161] "Multi-modality AI" is an AI technology that integrates multiple different data modalities (e.g., text, images, audio), analyzes them comprehensively, and makes highly accurate task suggestions and predictions.

[0162] A "suggested task" is the next action or task that the AI ​​identifies based on user input and acquired data.

[0163] "Noise reduction" is a process for removing unnecessary information and data in image preprocessing to improve image quality.

[0164] "Resolution optimization" is a process that optimizes the fine details of an image to improve the accuracy of character recognition.

[0165] "User notification" means the notification method used by the system to inform the user of required information or tasks, including pop-up windows and alerts.

[0166] "Task execution" means actually performing the proposed task selected by the user, which includes launching an application and registering data.

[0167] "Price comparison" refers to the process of collecting and comparing price information for a specific product at different sales locations and times to provide users with the best purchasing options.

[0168] "Review display" refers to displaying ratings and impressions posted by other users about a specific product, allowing users to refer to other people's opinions before purchasing.

[0169] The system for implementing the present invention includes the following detailed processes to support users' shopping activities.

[0170] The programs implemented on the server and terminal first include a means for acquiring and preprocessing camera images. The terminal's camera is activated, and the user takes a photo of the product or shelf information. The captured image is temporarily saved on the terminal and then sent to the server. The server uses the image processing library OpenCV (cv2) to remove noise and optimize the resolution of the received image data. After preprocessing, the image is input into the character recognition engine Tesseract (pytesseract), and character information such as the product name, price, and features is extracted as text data.

[0171] The extracted text data and surrounding contextual information are analyzed by a multi-modality AI on the server to identify new tasks. This AI model performs cloud-based analysis using the Google Cloud Vision API to generate optimal suggested tasks for the user (e.g., comparing product prices or displaying reviews) and sort them by priority. The resulting task list is notified to the user and sent to the device. Notifications are displayed as pop-up windows or alerts on the device.

[0172] The user checks the notification and selects one of the suggested tasks. For example, if they select the task of comparing prices of a specific product with other stores, the device will launch a web browser and display the relevant search page. For other tasks, such as displaying product reviews, the device will display review information from appropriate sites. This process allows the user to efficiently proceed with their shopping.

[0173] Furthermore, the results of the task execution are sent to the server as feedback and saved as learning data to improve the accuracy of the next task proposal. The server uses this feedback to continuously improve the accuracy of the generative AI model.

[0174] Specific prompt examples:

[0175] Prompt after taking a photo of the receipt:

[0176] Your receipt information has been extracted. What do you want to do with this information? Choose from the following options:

[0177] 1. Register for a household accounting app

[0178] 2. Save the coupon

[0179] 3. Compare prices with other stores

[0180] Product comparison prompt:

[0181] Enter the name of the product you want to search for:

[0182] The system allows users to efficiently gather product information and compare prices based on captured images, enabling them to make the best purchasing decision. Specific hardware components include smartphones and smart glasses, while software components include OpenCV, Tesseract, Google Cloud Vision API, and multi-modality artificial intelligence.

[0183] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0184] Step 1: Acquire camera images

[0185] The device activates the camera, and the user takes a picture of the product or shelf information. The input is image data acquired from the device's camera, and the output is an image file temporarily stored in the device. This image file is used in the next processing step.

[0186] Step 2: Image preprocessing

[0187] The device sends the saved image file to the server. The server performs noise removal and resolution optimization on the received image data. The input is the image data sent from the device, and the output is a preprocessed image that has been noise removed and resolution optimized. This process uses an image processing library called OpenCV.

[0188] Step 3: Character Recognition

[0189] The server inputs the preprocessed image data into the OCR engine (Tesseract). The OCR engine recognizes the characters in the image and extracts them as text data. The input is the preprocessed image data, and the output is the extracted text data. Specifically, it detects the outlines of the characters in the image and recognizes them as characters.

[0190] Step 4: Analyze text and context

[0191] The server inputs the text information extracted by the OCR engine and the surrounding background information into a multi-modality artificial intelligence. The AI ​​analyzes this information and suggests related tasks to the user (for example, comparing product prices or displaying reviews). The input is the extracted text data and background information, and the output is a task list as the analysis result. Specifically, it extracts product features from the text data and generates related tasks based on this.

[0192] Step 5: Notify users

[0193] The server sends the generated task list to the terminal and notifies the user. The notification is displayed on the terminal as a pop-up window or alert. The input is the task list as the analysis result, and the output is the notification to the user. Specifically, it lists tasks in order of priority based on specific conditions and presents this to the user.

[0194] Step 6: User Task Selection

[0195] The user selects one of the suggested tasks. The input is the task list presented as a notification, and the output is the user's selection. The specific behavior involves selecting a task through a user interface.

[0196] Step 7: Execute the task

[0197] The device executes the selected task based on the user's input. For example, if a user wants to compare the prices of a particular product with other stores, the device will launch a web browser and display the relevant page. If a user wants to view product reviews, the device will display the review information from the appropriate website. The input is the user's selection, and the output is the result of each task. Specific operations include launching an application or displaying data according to the selected task.

[0198] Step 8: Feedback on results

[0199] The results of the task execution are sent from the device to the server. The server stores this feedback in a database and uses it as training data to improve the accuracy of the next task suggestion. The input is the task execution result, and the output is a database entry for the next training. Specific operations include evaluating the execution results and recording them.

[0200] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0201] An embodiment of the present invention is described below. The present invention provides a system that combines an OCR application that uses a camera to recognize characters, analyzes peripheral information, and suggests and executes the next task to be performed with an emotion engine that recognizes the user's emotions.

[0202] 1. Camera image acquisition and preprocessing

[0203] The device activates the camera and captures the image taken by the user. For example, the user launches an application and takes a picture of a receipt or document with the camera. The image is temporarily stored on the device and then sent to the server. The server preprocesses the received image, removing noise and optimizing the resolution to improve the accuracy of character recognition.

[0204] Examples:

[0205] The user takes a photo of the receipt, and the image is sent from the device to a server where noise is removed and resolution is adjusted.

[0206] 2. Character Recognition Processing

[0207] The server inputs the preprocessed image data into a character recognition engine. The OCR engine (e.g., an open-source OCR tool or a cloud-based OCR API) recognizes the characters in the image and extracts them as text data. The extracted text is then checked for spelling and grammar before being formatted into accurate data.

[0208] Examples:

[0209] The server extracts text information such as the product name, price, and store name from the receipt image.

[0210] 3. Analysis of surrounding information and task proposal

[0211] The server inputs the text information extracted by the OCR engine and the surrounding background information into a multi-modality AI. The AI ​​analyzes this information and identifies the most relevant tasks for the user. As a result of the analysis, a task list is generated and sorted by priority. This task list is then converted into a format that can be displayed to the user and sent to the device.

[0212] Examples:

[0213] The server uses the extracted receipt information and background information to suggest tasks such as "register this expenditure in a household accounting app" or "save a coupon."

[0214] 4. User Emotion Recognition

[0215] The device detects the user's facial expressions and tone of voice through the camera and microphone, and recognizes the user's emotions using an emotion engine. For example, when a user takes a photo of a receipt, the camera detects the user's smile. This information is sent from the device to the server.

[0216] Examples:

[0217] When a user takes a photo of a receipt using their smartphone camera, the emotion engine recognizes the user's smile (happiness) emotion.

[0218] 5. Emotion-Based Task Adjustment

[0219] The server re-evaluates the task list based on the emotional information recognized by the emotion engine and prioritizes tasks that suit the user's emotions. For example, if the user looks tired, it will prioritize easy tasks.

[0220] Examples:

[0221] The emotion engine detects when the user has a tired expression, and the server prioritizes and suggests a simplified task such as "immediately record this expenditure in a household accounting app."

[0222] 6. User Notification and Task Execution

[0223] The device notifies the user of the task list sent from the server. The notification is displayed in the form of a pop-up window or an alert. When the user selects one of the suggested tasks, the device executes that task. For example, it may register data in a household accounting app or launch an email sending app.

[0224] Examples:

[0225] When the user selects "Register in household accounting app," the device opens the household accounting app and enters the receipt information.

[0226] 7. Feedback of results

[0227] Once the task is completed, the device sends the result (success, failure, feedback) to the server. The server stores the received feedback in a database and uses it as training data for the AI ​​model to improve the accuracy of the next task suggestions. Emotion recognition results are also stored as learning data and used to improve the user experience.

[0228] Examples:

[0229] If the data is successfully registered in the household accounting app, the information is sent to the server and used for analyzing receipt information the next time. Also, if the user smiles with satisfaction after completing the task, the emotional information is also saved.

[0230] This embodiment allows users to efficiently handle a variety of tasks with a single OCR application, eliminating the need to use multiple dedicated applications. Furthermore, the combination of emotion engines enables optimal task suggestions based on the user's emotional state, resulting in a more intuitive and satisfying user experience.

[0231] The processing flow will be explained below.

[0232] Step 1:

[0233] The user launches the OCR application and switches to camera mode. The user presses the capture button to capture an image.

[0234] Step 2:

[0235] The captured image is temporarily stored on the device and then sent to the server.

[0236] Step 3:

[0237] The server decodes the received image and performs pre-processing such as adjusting the resolution and removing noise.

[0238] Step 4:

[0239] The server inputs the preprocessed image data into a character recognition engine, and the OCR engine extracts character information from the image.

[0240] Step 5:

[0241] The server performs spelling and grammar checks on the extracted text information and formats it into accurate text data.

[0242] Step 6:

[0243] The server inputs the text information and surrounding background information into the multi-modality artificial intelligence, and the AI ​​begins its analysis.

[0244] Step 7:

[0245] Based on the analysis results, the server's AI generates suggested tasks that are most relevant to the user and creates a task list sorted by priority.

[0246] Step 8:

[0247] The server formats the generated task list and sends it to the terminal for notification to the user.

[0248] Step 9:

[0249] The device displays the task list to the user as a popup window or alert, and the user selects one of the suggested tasks.

[0250] Step 10:

[0251] The device executes the corresponding task based on the user's selection, for example, launching a household accounting app and entering text data.

[0252] Step 11:

[0253] The device detects the user's facial expressions and tone of voice through the camera and microphone, and recognizes the user's emotions using an emotion engine.

[0254] Step 12:

[0255] The device sends the user's emotional information to the server, which then reevaluates and adjusts the task list based on the emotional information.

[0256] Step 13:

[0257] The server transmits the task list readjusted based on the emotion information to the terminal, and the terminal notifies the user again.

[0258] Step 14:

[0259] If the user selects the suggested task again, the device will execute the adjusted task, for example, prioritizing the execution of the easier task.

[0260] Step 15:

[0261] After the task is completed, the device sends the result (success, failure, feedback) to the server.

[0262] Step 16:

[0263] The server stores the results it receives in a database and uses them as training data for the AI ​​model to improve the accuracy of the next task proposal, including the emotion recognition results.

[0264] Example 2

[0265] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0266] In modern society, many users need to quickly and accurately manage various documents and payment records in their daily lives. Conventional applications using optical character recognition (OCR) technology only provide character recognition and simple data conversion, but lack the ability to manage tasks or suggest tasks based on the user's emotional state. In particular, task suggestions that take into account the user's mental and emotional state are crucial for improving the user experience. Therefore, there is a need for a system that allows users to efficiently manage documents and receive appropriate task suggestions based on their emotional state.

[0267] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0268] In this invention, the server includes means for activating a camera to acquire an image, means for preprocessing the acquired image to remove noise and optimize resolution, means for using a character recognition engine to extract character information from the preprocessed image, means for analyzing the extracted character information and surrounding background information to identify a next suggested task to be performed, means for using an emotion engine to recognize the user's emotions, means for reevaluating the task list based on the recognized emotion information, means for notifying the user of the suggested tasks, means for executing the suggested tasks based on the user's input, and means for recording the execution results and emotion information and evaluating them to improve the accuracy of the next suggestion. This enables the user to efficiently and accurately perform a series of document management and task suggestions, and further enables the user to receive optimal task suggestions adapted to their emotional state.

[0269] "Means for activating the camera" refers to a device or software that has the functionality to allow a user to activate the camera and take an image by operating an application.

[0270] "Means for preprocessing images" refers to devices or software that have the function of removing noise and optimizing resolution from acquired images, and performing processing to improve the accuracy of character recognition.

[0271] A "character recognition engine" is software or algorithms that use OCR technology to extract character information from pre-processed image data.

[0272] "Surrounding background information" refers to information other than textual information contained in an image, such as location information or user history information, and is additional data used to suggest tasks.

[0273] An "emotion engine" is software or algorithms that use a camera or microphone to analyze facial expressions and tone of voice in order to recognize a user's emotional state (e.g., joy or fatigue).

[0274] A "suggested task" is a suggestion to the user about the next action or process to be taken based on the text information and surrounding information analyzed by the system.

[0275] The "means for reevaluating the task list" refers to a function that readjusts the priority of an already generated task list based on the user's emotional information and performs the process of suggesting the most suitable task list to the user.

[0276] A "notification mechanism" is any device or software capable of displaying information in the form of a pop-up window, alert, or other form to inform the user of a proposed task.

[0277] A "means for performing a task" is any device or software that has the functionality to actually perform the proposed task selected by the user.

[0278] "Means for recording execution results" refers to devices or software that have the ability to record the results (success, failure, feedback) after a task is completed and store them in a database to improve the accuracy of suggestions next time.

[0279] An embodiment of the present invention is described below. The present invention provides a system that combines an OCR application that uses a camera to recognize characters, analyzes peripheral information, and suggests and executes the next task to be performed with an emotion engine that recognizes the user's emotions.

[0280] Camera image acquisition and preprocessing

[0281] The device activates the camera and acquires an image taken by the user. For example, the user launches an application and takes a picture of a receipt or document with the camera. This image is temporarily stored on the device and then sent to the server. The server then preprocesses the received image, removing noise and optimizing the resolution to improve the accuracy of character recognition. Specifically, it uses noise reduction algorithms such as Gaussian Blur and image resolution optimization techniques.

[0282] Example: A user takes a photo of a receipt, the image is sent from the device to a server, which removes noise from the image and scales it to the appropriate resolution.

[0283] Character Recognition Processing

[0284] The server inputs the preprocessed image data into a character recognition engine. An OCR engine (such as the open-source Tesseract or the cloud-based Google Cloud Vision API) recognizes the characters in the image and extracts them as text data. The extracted text is then checked for spelling and grammar to ensure accurate data is generated.

[0285] Example: The server extracts text information such as "product name," "price," and "store name" from a receipt image and performs spell checking and grammar correction.

[0286] Peripheral information analysis and task suggestions

[0287] The server inputs the text information extracted by the OCR engine and the surrounding background information into a multi-modality AI. The AI ​​analyzes this information and identifies the most relevant tasks for the user. As a result of the analysis, a task list is generated and sorted by priority. The generated task list is converted into a notification format and sent to the device.

[0288] Example: The server uses the extracted receipt information and background information to identify tasks such as "record this expenditure in a household accounting app" or "save a coupon" and create a list.

[0289] User emotion recognition

[0290] The device detects the user's facial expressions and tone of voice through the camera and microphone, and recognizes the user's emotions using an emotion engine. For example, it uses machine learning models for emotion recognition (Microsoft Face API or Emotion API). The recognized emotion information is sent from the device to a server.

[0291] Example: When a user takes a picture of a receipt using the smartphone camera, the emotion recognition engine recognizes the user's smile (happiness) emotion and sends that information to the server.

[0292] Emotion-based task adjustment

[0293] The server reevaluates the task list based on the emotional information received. It then adjusts the priority of tasks based on the recognized emotional information and suggests the best possible tasks for the user. For example, if the user looks tired, the list is adjusted to prioritize easier tasks.

[0294] Example: If the emotion engine detects that the user has a tired expression, the server will prioritize and suggest simplified tasks.

[0295] User notification and task execution

[0296] The device notifies the user of the task list sent from the server. The notification is displayed in the form of a pop-up window or an alert, and when the user selects one of the suggested tasks, the device executes that task. For example, it may register data in a household accounting app or launch an email sending app.

[0297] Example: When a user selects "Register in household accounting app," the device opens the household accounting app and enters receipt information.

[0298] Feedback of results

[0299] Once the task is completed, the device sends the result (success, failure, feedback) to the server. The server stores the received feedback in a database and uses it as training data for the AI ​​model to improve the accuracy of the next task suggestion. The server also stores the emotion recognition results as learning data to help improve the user experience.

[0300] Example: When data is successfully registered in a household accounting app, the information is sent to the server and used for analyzing receipt information the next time. If the user smiles with satisfaction after completing a task, the emotional information is also stored on the server.

[0301] Example prompts to input to the generative AI model

[0302] "Please explain in detail the system that analyzes receipt information and suggests appropriate tasks for the user."

[0303] "Please explain the specific steps in the process flow of a system that recognizes a user's emotions and adjusts tasks based on those emotions."

[0304] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0305] Step 1:

[0306] The device starts the camera and acquires the image taken by the user. The user starts the application and takes a picture of a receipt or document with the camera. As a result, the image data is temporarily stored in the device's local memory. The input data is the camera image, and the output data is an image file stored on the device.

[0307] Step 2:

[0308] The device sends the captured image to the server. Here, the device transfers the image data to the server using a secure communication protocol (e.g., HTTPS). The input data is the image file stored in the device, and the output data is the image file sent to the server. Specifically, data is transferred using an HTTP request.

[0309] Step 3:

[0310] The server preprocesses the images received. Preprocessing includes noise removal (for example, applying Gaussian Blur) and resolution optimization. The input data is the image file received by the server, and the output data is the preprocessed image data. Specifically, the image processing algorithm is applied using Python's OpenCV library.

[0311] Step 4:

[0312] The server inputs the preprocessed image data into a character recognition engine. An OCR engine (e.g., Tesseract) is used to recognize the characters in the image and extract them as text data. The input data is the preprocessed image data, and the output data is character string data (text). Specifically, the OCR engine is called to perform image analysis and the recognition results are obtained as text.

[0313] Step 5:

[0314] The server performs spelling and grammar checks on the extracted text data. It uses natural language processing tools (e.g., NLTK or SpaCy) to analyze the text data and format it into accurate data. The input data is text data obtained from the OCR engine, and the output data is accurately formatted text data. Specifically, it performs grammar and spelling checks using regular expressions and machine learning models.

[0315] Step 6:

[0316] The server inputs the text information extracted by the OCR engine and the surrounding background information into a multi-modality AI. The multi-modality AI analyzes this information and identifies tasks that are highly relevant to the user. The input data is precisely formatted text data and surrounding information, and the output data is a task list sorted by priority. Specifically, the data is input into the AI ​​model and task relevance is calculated.

[0317] Step 7:

[0318] The server converts the generated task list into a notification format and sends it to the terminal. The input data is the task list, and the output data is in a notification data format (e.g., JSON format). Specific operations include data formatting and sending operations.

[0319] Step 8:

[0320] The terminal notifies the user of the task list sent from the server. The notification is displayed as a pop-up window or an alert. The input data is the notification data received from the server, and the output data is the notification visually displayed to the user. Specifically, the notification is displayed using GUI (Graphical User Interface) elements.

[0321] Step 9:

[0322] The user selects one of the proposed tasks. The input data is the presented task list, and the output data is the task selected by the user. The selection is made by the user's action.

[0323] Step 10:

[0324] The device executes a task selected by the user. For example, it registers data in a household accounting app or launches an email sending app. The input data is the task selected by the user, and the output data is the task completion status (success or failure). The specific operation is to call the API of the corresponding application.

[0325] Step 11:

[0326] When the task is completed, the device sends the result (success, failure, feedback) to the server. The input data is the task completion status, and the output data is the feedback data. Specifically, the feedback is sent to the server using an HTTP request.

[0327] Step 12:

[0328] The server stores the received feedback in a database and uses it to improve the accuracy of the next task suggestion. It also stores the emotion recognition results and uses them to improve the user experience. The input data is the feedback data and the emotion recognition data, and the output data is the updated database entry. Specifically, the system stores and updates data using database queries.

[0329] (Application example 2)

[0330] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0331] In modern brick-and-mortar stores, store staff must manage multiple tasks simultaneously, requiring efficient operations. They are also required to provide high-quality service to customers, making store management more complex. Conventional systems suggest tasks without considering the user's emotional state, which can increase the burden on staff and reduce customer satisfaction. This invention provides a system that recognizes the user's emotional state and appropriately suggests and executes tasks based on that state, thereby improving the efficiency of brick-and-mortar stores and the customer experience.

[0332] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0333] In this invention, the server includes means for extracting text information from an image acquired by a camera using a character recognition engine, means for analyzing the extracted text information and surrounding background information to identify the next suggested task to be performed, means for detecting the user's facial expression and tone of voice to recognize emotions, and means for sorting the suggested tasks in order of priority based on the emotional information, thereby making it possible to suggest and execute appropriate tasks based on the emotional states of store staff and customers.

[0334] A "character recognition engine" is a device or program that analyzes and extracts character information from images acquired by a camera.

[0335] "Surrounding background information" is a general term for accompanying data and environmental information related to text information.

[0336] A "proposed task" is a proposal for the next work to be done based on the analyzed and parsed information.

[0337] "User notification means" refers to the means by which the proposed task is communicated to the user, displayed as a notification or alert.

[0338] A "task executor" is a processing mechanism for executing a suggested task based on user input.

[0339] "Facial expression detection" is a process that uses a camera to analyze a user's facial expressions and recognize their emotions.

[0340] "Tone of voice detection" is a process that uses a microphone to analyze the tone of a user's voice and recognize their emotions.

[0341] "Emotion recognition" is the process of identifying a user's emotions from facial expressions and tone of voice.

[0342] The "means for sorting by priority" is a processing function that rearranges the order of proposed tasks based on their importance and ease of execution.

[0343] "Multi-modality artificial intelligence" is an artificial intelligence system that combines and analyzes multiple data formats (for example, text information and background information) to provide users with appropriate information.

[0344] This invention uses a "smart store assistant" system in a physical store to efficiently and effectively suggest tasks to store staff and customers and assist them in carrying them out. A specific embodiment of this system will be described below.

[0345] First, the device uses a camera to scan the barcode or label of a product in a physical store. This device can be a smartphone or smart glasses. The camera is activated and the captured image is temporarily stored on the device. The image is then sent to a server for pre-processing. This pre-processing involves removing noise from the image and adjusting its resolution to improve the accuracy of character recognition by the OCR engine.

[0346] Next, the server inputs the preprocessed image data into an OCR engine, such as Tesseract OCR or Google Cloud Vision API. This extracts the text information from the image and records it as text data. At this stage, information such as the product name, price, and expiration date are obtained.

[0347] The extracted text information is input into a multi-modality AI system. The server analyzes the text information in combination with the surrounding contextual information to identify tasks to be proposed. For example, this could include suggesting restocking of low-stock items or proposing promotions for specific products.

[0348] The device detects the user's facial expressions and tone of voice through a camera and microphone, and analyzes them with an emotion recognition engine. The recognized emotion information is sent to the server and reflected in task suggestions. For example, if the user is tired, the device will prioritize simple tasks to reduce the user's burden.

[0349] Finally, the server sends the prioritized task list to the device. The device notifies the user of the task list through a pop-up window or an alert. When the user selects one of the suggested tasks, the task is automatically executed. For example, it may send a stock replenishment request or start a sales promotion campaign. Once the task is completed, the device feeds back the results to the server. The server stores this in a database and uses it to improve the accuracy of future suggestions.

[0350] For example, here's a prompt to notify you when an item is low in stock:

[0351] Scan the receipt image.

[0352] Important Task: This item is low on stock and needs to be restocked. Would you like to submit a restock request?

[0353] This system will reduce the burden on store staff in physical stores, enable them to carry out their work more efficiently, and improve customer satisfaction.

[0354] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0355] Step 1:

[0356] The device uses a camera to scan products or labels in a physical store. The user picks up the device (smartphone or smart glasses) and points the camera at the target product to capture an image. In this step, the camera is activated and the captured image is temporarily saved in the device. The image is then sent to the server. The input is the image of the product or label, and the output is the image data sent to the server.

[0357] Step 2:

[0358] The server preprocesses the received image data. Preprocessing includes noise removal and resolution adjustment. Specifically, the server converts the image to grayscale and applies Gaussian blur to remove noise. It also adjusts the resolution appropriately to improve the accuracy of character recognition. The input is the received raw image data, and the output is preprocessed, high-quality image data.

[0359] Step 3:

[0360] The server inputs the preprocessed image data into the OCR engine. The OCR engine (for example, Tesseract OCR or Google Cloud Vision API) analyzes the character information in the image and extracts it as text data. In this step, the server calls the OCR engine to perform the analysis and saves the obtained text data on the server. The input is the preprocessed image data, and the output is text data containing character information.

[0361] Step 4:

[0362] The server inputs the extracted text information and surrounding background information into a multi-modality AI for analysis. This identifies suggested tasks that are highly relevant to the user. The server then integrates the text information and background information and analyzes it using an AI model. As a result, a list of suggested tasks is generated. The input is text information and background information, and the output is a list of suggested tasks.

[0363] Step 5:

[0364] The device detects the user's facial expressions and tone of voice through a camera and microphone, and analyzes them using an emotion recognition engine. When the user operates the device, the camera captures the user's facial expressions and the microphone records the user's tone of voice. This data is sent from the device to a server and analyzed by the emotion recognition engine. The input is facial expression and voice data, and the output is recognized emotional information.

[0365] Step 6:

[0366] The server re-evaluates the proposed task list based on the emotion information and sorts it in order of priority. The server receives the emotion recognition results and adjusts the task priority according to the user's state. For example, if the user is tired, it prioritizes easier tasks. The input is the proposed task list and emotion information, and the output is a task list sorted in order of priority.

[0367] Step 7:

[0368] The terminal notifies the user of a priority-ordered task list sent from the server. The terminal displays the task list in a pop-up window or alert format, allowing the user to select from the list. When the user selects a specific task, that task is executed. The input is a task list sorted by priority, and the output is a notification to the user and the execution of the task.

[0369] Step 8:

[0370] After completing a task, the device sends the result (success, failure, feedback) to the server. The server stores the received feedback in a database and uses it as training data for the AI ​​model to improve the accuracy of the next task proposal. The input is the task execution result and feedback, and the output is the feedback information stored as learning data.

[0371] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0372] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0373] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0374] [Second embodiment]

[0375] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0376] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0377] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0378] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0379] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0380] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0381] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0382] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0383] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0384] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0385] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0386] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0387] An embodiment of the present invention will be described below. The present invention provides an OCR application that uses a camera to recognize characters, analyzes surrounding information, and suggests and executes the next task to be performed.

[0388] 1. Camera image acquisition and preprocessing

[0389] The device activates the camera and captures the image taken by the user. For example, the user activates a scanner application and takes a picture of a receipt or document with the camera. The image is temporarily stored on the device and then sent to the server. The server preprocesses the received image, removing noise and optimizing the resolution to improve the accuracy of character recognition.

[0390] Examples:

[0391] The user takes a photo of the receipt, and the image is sent from the device to a server where noise is removed and resolution is adjusted.

[0392] 2. Character Recognition Processing

[0393] The server inputs the preprocessed image data into a character recognition engine. The OCR engine (e.g., an open-source OCR tool or a cloud-based OCR API) recognizes the characters in the image and extracts them as text data. The extracted text is then checked for spelling and grammar before being formatted into accurate data.

[0394] Examples:

[0395] The server extracts text information such as the product name, price, and store name from the receipt image.

[0396] 3. Analysis of surrounding information and task proposal

[0397] The server inputs the text information extracted by the OCR engine and the surrounding background information into a multi-modality AI. The AI ​​analyzes this information and identifies the most relevant tasks for the user. As a result of the analysis, a task list is generated and sorted by priority. This task list is then converted into a format that can be displayed to the user and sent to the device.

[0398] Examples:

[0399] The server uses the extracted receipt information and background information to suggest tasks such as "register this expenditure in a household accounting app" or "save a coupon."

[0400] 4. User Notification and Task Execution

[0401] The device notifies the user of the task list sent from the server. The notification is displayed in the form of a pop-up window or an alert. When the user selects one of the suggested tasks, the device executes that task. For example, it may register data in a household accounting app or launch an email sending app.

[0402] Examples:

[0403] When the user selects "Register in household accounting app," the device opens the household accounting app and enters the receipt information.

[0404] 5. Feedback of results

[0405] Once the task is completed, the device sends the result (success, failure, feedback) to the server, which stores the received feedback in a database and uses it as training data for the AI ​​model to improve the accuracy of the next task suggestions.

[0406] Examples:

[0407] If the data is successfully registered in the household accounting app, the information is sent to the server and used for analyzing receipt information the next time.

[0408] This embodiment allows users to efficiently handle various tasks with one OCR application, eliminating the need to use multiple dedicated applications. Furthermore, the use of multi-modality artificial intelligence improves the accuracy of suggested tasks and makes user operation easier and more intuitive.

[0409] The processing flow will be explained below.

[0410] Step 1:

[0411] The user launches the OCR application and switches to camera mode. The user presses the capture button to capture an image.

[0412] Step 2:

[0413] The captured image is temporarily stored on the device and then sent to the server.

[0414] Step 3:

[0415] The server decodes the received image into a file format and performs preprocessing such as adjusting the resolution and removing noise.

[0416] Step 4:

[0417] The server inputs the preprocessed image data into a character recognition engine, and the OCR engine extracts character information from the image.

[0418] Step 5:

[0419] The server performs spelling and grammar checks on the extracted text information and formats it into accurate text data.

[0420] Step 6:

[0421] The server inputs the text information and surrounding background information into the multi-modality artificial intelligence, and the AI ​​begins its analysis.

[0422] Step 7:

[0423] Based on the analysis results, the server's AI generates suggested tasks that are most relevant to the user and creates a task list sorted by priority.

[0424] Step 8:

[0425] The server formats the generated task list and sends it to the terminal for notification to the user.

[0426] Step 9:

[0427] The device displays the task list to the user as a popup window or alert, and the user selects one of the suggested tasks.

[0428] Step 10:

[0429] The device executes the corresponding task based on the user's selection, for example, launching a household accounting app and entering text data.

[0430] Step 11:

[0431] The server receives the task execution results (success, failure, feedback) from the terminal and stores them in a database.

[0432] Step 12:

[0433] The server uses the saved result data as learning data for the AI ​​and updates the model to improve the accuracy of the next proposed task.

[0434] Example 1

[0435] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0436] Conventional OCR applications have insufficient character recognition accuracy, often requiring users to manually correct information. Furthermore, information analysis and task suggestions after character recognition are ineffective, resulting in poor user convenience. Furthermore, the accuracy of the suggested tasks is low, and there is a lack of a learning mechanism that utilizes feedback.

[0437] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0438] In this invention, the server includes means for temporarily saving images acquired from the camera in an internal storage device, means for transmitting images from the internal storage device to the server and performing noise reduction and resolution optimization, means for using a character recognition engine to extract character information from the noise-removed and resolution-optimized images, means for analyzing the extracted character information and surrounding background information using multi-modality artificial intelligence and identifying a suggested next task to be performed, means for notifying the user of the suggested task, means for executing the suggested task based on user input, and means for sending feedback after task execution to the server and storing the feedback in a database to use as training data. This enables improved character recognition accuracy, effective information analysis and task suggestions, and a learning mechanism using feedback.

[0439] A "camera" is a device for taking still images and videos.

[0440] "Internal storage" means a storage medium within a device that is used to store data temporarily or permanently.

[0441] A "server" is a computer system that provides services to other computers on a network.

[0442] "Noise reduction" is the process of removing unwanted interference and noise from image and audio data.

[0443] "Resolution optimization" is a process of adjusting the image so that the details can be displayed clearly.

[0444] A "character recognition engine" is software or an algorithm that identifies characters in an image and extracts them as text data.

[0445] "Multi-modality artificial intelligence" is artificial intelligence that integrates, understands, and analyzes data in multiple different formats (e.g., text and images).

[0446] "Analysis" is the process of examining data in detail to understand its meaning and structure.

[0447] A "suggested task" is a specific action or task that is shown to the user to take next.

[0448] "Notification means" refers to a method or device for notifying a user of information.

[0449] "Feedback" refers to reporting the results of a task and user evaluation information to the server, and is used for future improvements.

[0450] A "database" is a system for efficiently storing, managing, and searching data.

[0451] An embodiment of the present invention will be described in detail. The invention provides an OCR application that uses a camera to recognize characters, analyzes surrounding information, and proposes and executes the next task to be performed. The following describes the flow of data processing and data calculation using each hardware and software.

[0452] 1. Camera image acquisition and preprocessing

[0453] The device activates the camera and acquires the image taken by the user. For example, the user activates a scanner application and takes a picture of a receipt or document with the camera. This image is temporarily stored in the device's internal storage and then sent to the server. The server preprocesses the received image, removing noise and optimizing the resolution to improve the accuracy of character recognition. For example, it uses the image processing library "OpenCV" to remove noise and optimizes the resolution using the "PhotoShop API."

[0454] Examples:

[0455] The user takes a photo of the receipt, and the image is sent from the device to a server where noise is removed and resolution is adjusted.

[0456] 2. Character Recognition Processing

[0457] The server inputs the preprocessed image data into a character recognition engine. The OCR engine used can be the open source "Tesseract" or the cloud-based "Google Cloud Vision API." The server calls the OCR engine, recognizes the characters in the image, and extracts them as text data. The extracted text is then checked for spelling and grammar using a natural language processing tool (such as "NLTK" or "SpaCy"), and formatted into accurate text data.

[0458] Examples:

[0459] The server extracts text information such as the product name, price, and store name from the receipt image.

[0460] 3. Analysis of surrounding information and task proposal

[0461] The server inputs the text information extracted by the OCR engine and the surrounding background information into a multi-modality AI (such as GPT-3 or BERT). The AI ​​analyzes this information and identifies the most relevant tasks for the user. As a result of the analysis, a task list is generated and sorted by priority. This task list is then converted into a format that can be displayed to the user and sent to the device.

[0462] Examples:

[0463] The server uses the extracted receipt information and background information to suggest tasks such as "register this expenditure in a household accounting app" or "save a coupon."

[0464] 4. User Notification and Task Execution

[0465] The device notifies the user of the task list sent from the server. The notification is displayed in the form of a pop-up window or an alert. When the user selects one of the suggested tasks, the device executes that task. Specifically, it registers data in a household accounting app or launches an email sending app.

[0466] Examples:

[0467] When the user selects "Register in household accounting app," the device opens the household accounting app and inputs the receipt information.

[0468] 5. Feedback of results

[0469] Once the task is completed, the device sends the result (success, failure, feedback) to the server, which stores the received feedback in a database and uses it as training data for the AI ​​model to improve the accuracy of the next task suggestions.

[0470] Examples:

[0471] If the data is successfully registered in the household accounting app, the information is sent to the server and used for analyzing receipt information the next time.

[0472] Prompt Sentence Examples

[0473] In a system for implementing the invention, more specific task suggestions can be made by using prompt sentences such as the following to input to a generative AI model:

[0474] example:

[0475] "I took a photo of my receipt. Can you give me some suggestions to help me manage my household finances and coupons?"

[0476] The above is an embodiment of the present invention. This embodiment allows users to efficiently handle a variety of tasks with a single OCR application, eliminating the need to use multiple dedicated applications. Furthermore, by utilizing multi-modality artificial intelligence, the accuracy of the suggested tasks is improved, and user operation becomes simpler and more intuitive.

[0477] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0478] Step 1:

[0479] A user starts a scanner application on their device and uses the camera to take a picture of a receipt or document. At this time, the device's camera captures the image data and temporarily stores it in the device's internal storage. The input is the receipt image taken by the user, and the output is the image data stored in the internal storage.

[0480] Step 2:

[0481] The device sends image data stored in its internal storage device to the server. The image data is sent over the Internet using a secure communication protocol such as HTTPS. The input is the image data in the internal storage device, and the output is the image data sent to the server.

[0482] Step 3:

[0483] The server analyzes the image data it receives and performs preprocessing to remove noise and optimize resolution. Specifically, it removes noise using the image processing library "OpenCV" and optimizes resolution using "PhotoShop API." The input is the image data sent to the server, and the output is image data with noise removed and resolution optimized.

[0484] Step 4:

[0485] The server inputs the preprocessed image data into a character recognition engine. The OCR engine used is "Tesseract" or "Google Cloud Vision API." The server uses the OCR engine to recognize characters in the image and extract them as text data. The input is noise-removed and resolution-optimized image data, and the output is the extracted text data.

[0486] Step 5:

[0487] The server uses natural language processing tools (such as NLTK and SpaCy) to check the spelling and grammar of the extracted text data, which then converts it into accurate text data. The input is text data obtained from the OCR engine, and the output is accurate text data that has been checked for spelling and grammar.

[0488] Step 6:

[0489] The server inputs the text data extracted by the OCR engine and the surrounding context into a multi-modality AI model (e.g., GPT-3 or BERT). The AI ​​model analyzes this information and identifies tasks that are highly relevant to the user. The input is the formatted text data and context, and the output is a list of suggested tasks.

[0490] Step 7:

[0491] The server converts the generated task list into a notification format and sends it to the terminal. For example, it constructs the task list in JSON format and sends it to the terminal via the Internet. The input is the specified task list, and the output is the task list sent to the terminal.

[0492] Step 8:

[0493] The terminal analyzes the task list received from the server and notifies the user. The notification is displayed in the form of a pop-up window or an alert. The input is the task list sent from the server, and the output is the tasks to be notified to the user.

[0494] Step 9:

[0495] The user selects one of the proposed tasks, for example, registering data in a household accounting app. The input is the notified task list, and the output is the specific task selected.

[0496] Step 10:

[0497] The device executes the selected task. Specifically, it opens a household accounting app and automatically inputs the receipt information the user wants to register. The input is the selected task information, and the output is the executed task.

[0498] Step 11:

[0499] The terminal sends the result (success, failure, feedback) after completing the task to the server. The result data is sent to the server via the Internet. The input is the result of the task execution, and the output is the feedback data sent to the server.

[0500] Step 12:

[0501] The server stores the received feedback in a database. The feedback data is used as training data for the AI ​​model to improve the accuracy of the next task suggestion. The input is the feedback data sent from the device, and the output is the updated database information.

[0502] (Application example 1)

[0503] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0504] In physical stores, there is a need for support to enable users to quickly and accurately obtain product information and choose appropriate actions based on that information. However, current systems require manual information collection and analysis, which is extremely time-consuming. In addition, there is a lack of systems that allow users to easily compare prices and product reviews at other stores while shopping in a store. Therefore, technology is needed to enable users to shop efficiently and effectively.

[0505] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0506] In this invention, the server includes means for extracting text information from an image acquired by a camera using a character recognition engine, means for analyzing the extracted text information and surrounding background information and identifying a suggested next task to be performed, means for notifying the user of the suggested task, means for executing the suggested task based on the user's input, means for performing preprocessing such as image noise reduction and resolution optimization, means for performing cloud-based analysis, and means for suggesting tasks such as comparing product prices and displaying product reviews. This enables users to make optimal shopping decisions in real time based on product and shelf information captured by the camera.

[0507] A "character recognition engine" is software or hardware that recognizes characters from images acquired from a camera or other imaging device and extracts them as digital text data.

[0508] "Preprocessing" refers to processing of the captured image, such as noise removal and resolution optimization, to improve the accuracy of character recognition.

[0509] "Cloud-based analysis" is a technology that uses computing resources in a cloud environment to analyze and calculate data, enabling high-speed, large-scale data processing.

[0510] "Multi-modality AI" is an AI technology that integrates multiple different data modalities (e.g., text, images, audio), analyzes them comprehensively, and makes highly accurate task suggestions and predictions.

[0511] A "suggested task" is the next action or task that the AI ​​identifies based on user input and acquired data.

[0512] "Noise reduction" is a process for removing unnecessary information and data in image preprocessing to improve image quality.

[0513] "Resolution optimization" is a process that optimizes the fine details of an image to improve the accuracy of character recognition.

[0514] "User notification" means the notification method used by the system to inform the user of required information or tasks, including pop-up windows and alerts.

[0515] "Task execution" means actually performing the proposed task selected by the user, which includes launching an application and registering data.

[0516] "Price comparison" refers to the process of collecting and comparing price information for a specific product at different sales locations and times to provide users with the best purchasing options.

[0517] "Review display" refers to displaying ratings and impressions posted by other users about a specific product, allowing users to refer to other people's opinions before purchasing.

[0518] The system for implementing the present invention includes the following detailed processes to support users' shopping activities.

[0519] The programs implemented on the server and terminal first include a means for acquiring and preprocessing camera images. The terminal's camera is activated, and the user takes a photo of the product or shelf information. The captured image is temporarily saved on the terminal and then sent to the server. The server uses the image processing library OpenCV (cv2) to remove noise and optimize the resolution of the received image data. After preprocessing, the image is input into the character recognition engine Tesseract (pytesseract), and character information such as the product name, price, and features is extracted as text data.

[0520] The extracted text data and surrounding contextual information are analyzed by a multi-modality AI on the server to identify new tasks. This AI model performs cloud-based analysis using the Google Cloud Vision API to generate optimal suggested tasks for the user (e.g., comparing product prices or displaying reviews) and sort them by priority. The resulting task list is notified to the user and sent to the device. Notifications are displayed as pop-up windows or alerts on the device.

[0521] The user checks the notification and selects one of the suggested tasks. For example, if they select the task of comparing prices of a specific product with other stores, the device will launch a web browser and display the relevant search page. For other tasks, such as displaying product reviews, the device will display review information from appropriate sites. This process allows the user to efficiently proceed with their shopping.

[0522] Furthermore, the results of the task execution are sent to the server as feedback and saved as learning data to improve the accuracy of the next task proposal. The server uses this feedback to continuously improve the accuracy of the generative AI model.

[0523] Specific prompt examples:

[0524] Prompt after taking a photo of the receipt:

[0525] Your receipt information has been extracted. What do you want to do with this information? Choose from the following options:

[0526] 1. Register for a household accounting app

[0527] 2. Save the coupon

[0528] 3. Compare prices with other stores

[0529] Product comparison prompt:

[0530] Enter the name of the product you want to search for:

[0531] The system allows users to efficiently gather product information and compare prices based on captured images, enabling them to make the best purchasing decision. Specific hardware components include smartphones and smart glasses, while software components include OpenCV, Tesseract, Google Cloud Vision API, and multi-modality artificial intelligence.

[0532] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0533] Step 1: Acquire camera images

[0534] The device activates the camera, and the user takes a picture of the product or shelf information. The input is image data acquired from the device's camera, and the output is an image file temporarily stored in the device. This image file is used in the next processing step.

[0535] Step 2: Image preprocessing

[0536] The device sends the saved image file to the server. The server performs noise removal and resolution optimization on the received image data. The input is the image data sent from the device, and the output is a preprocessed image that has been noise removed and resolution optimized. This process uses an image processing library called OpenCV.

[0537] Step 3: Character Recognition

[0538] The server inputs the preprocessed image data into the OCR engine (Tesseract). The OCR engine recognizes the characters in the image and extracts them as text data. The input is the preprocessed image data, and the output is the extracted text data. Specifically, it detects the outlines of the characters in the image and recognizes them as characters.

[0539] Step 4: Analyze text and context

[0540] The server inputs the text information extracted by the OCR engine and the surrounding background information into a multi-modality artificial intelligence. The AI ​​analyzes this information and suggests related tasks to the user (for example, comparing product prices or displaying reviews). The input is the extracted text data and background information, and the output is a task list as the analysis result. Specifically, it extracts product features from the text data and generates related tasks based on this.

[0541] Step 5: Notify users

[0542] The server sends the generated task list to the terminal and notifies the user. The notification is displayed on the terminal as a pop-up window or alert. The input is the task list as the analysis result, and the output is the notification to the user. Specifically, it lists tasks in order of priority based on specific conditions and presents this to the user.

[0543] Step 6: User Task Selection

[0544] The user selects one of the suggested tasks. The input is the task list presented as a notification, and the output is the user's selection. The specific behavior involves selecting a task through a user interface.

[0545] Step 7: Execute the task

[0546] The device executes the selected task based on the user's input. For example, if a user wants to compare the prices of a particular product with other stores, the device will launch a web browser and display the relevant page. If a user wants to view product reviews, the device will display the review information from the appropriate website. The input is the user's selection, and the output is the result of each task. Specific operations include launching an application or displaying data according to the selected task.

[0547] Step 8: Feedback on results

[0548] The results of the task execution are sent from the device to the server. The server stores this feedback in a database and uses it as training data to improve the accuracy of the next task suggestion. The input is the task execution result, and the output is a database entry for the next training. Specific operations include evaluating the execution results and recording them.

[0549] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0550] An embodiment of the present invention is described below. The present invention provides a system that combines an OCR application that uses a camera to recognize characters, analyzes peripheral information, and suggests and executes the next task to be performed with an emotion engine that recognizes the user's emotions.

[0551] 1. Camera image acquisition and preprocessing

[0552] The device activates the camera and captures the image taken by the user. For example, the user launches an application and takes a picture of a receipt or document with the camera. The image is temporarily stored on the device and then sent to the server. The server preprocesses the received image, removing noise and optimizing the resolution to improve the accuracy of character recognition.

[0553] Examples:

[0554] The user takes a photo of the receipt, and the image is sent from the device to a server where noise is removed and resolution is adjusted.

[0555] 2. Character Recognition Processing

[0556] The server inputs the preprocessed image data into a character recognition engine. The OCR engine (e.g., an open-source OCR tool or a cloud-based OCR API) recognizes the characters in the image and extracts them as text data. The extracted text is then checked for spelling and grammar before being formatted into accurate data.

[0557] Examples:

[0558] The server extracts text information such as the product name, price, and store name from the receipt image.

[0559] 3. Analysis of surrounding information and task proposal

[0560] The server inputs the text information extracted by the OCR engine and the surrounding background information into a multi-modality AI. The AI ​​analyzes this information and identifies the most relevant tasks for the user. As a result of the analysis, a task list is generated and sorted by priority. This task list is then converted into a format that can be displayed to the user and sent to the device.

[0561] Examples:

[0562] The server uses the extracted receipt information and background information to suggest tasks such as "register this expenditure in a household accounting app" or "save a coupon."

[0563] 4. User Emotion Recognition

[0564] The device detects the user's facial expressions and tone of voice through the camera and microphone, and recognizes the user's emotions using an emotion engine. For example, when a user takes a photo of a receipt, the camera detects the user's smile. This information is sent from the device to the server.

[0565] Examples:

[0566] When a user takes a photo of a receipt using their smartphone camera, the emotion engine recognizes the user's smile (happiness) emotion.

[0567] 5. Emotion-Based Task Adjustment

[0568] The server re-evaluates the task list based on the emotional information recognized by the emotion engine and prioritizes tasks that suit the user's emotions. For example, if the user looks tired, it will prioritize easy tasks.

[0569] Examples:

[0570] The emotion engine detects when the user has a tired expression, and the server prioritizes and suggests a simplified task such as "immediately record this expenditure in a household accounting app."

[0571] 6. User Notification and Task Execution

[0572] The device notifies the user of the task list sent from the server. The notification is displayed in the form of a pop-up window or an alert. When the user selects one of the suggested tasks, the device executes that task. For example, it may register data in a household accounting app or launch an email sending app.

[0573] Examples:

[0574] When the user selects "Register in household accounting app," the device opens the household accounting app and enters the receipt information.

[0575] 7. Feedback of results

[0576] Once the task is completed, the device sends the result (success, failure, feedback) to the server. The server stores the received feedback in a database and uses it as training data for the AI ​​model to improve the accuracy of the next task suggestions. Emotion recognition results are also stored as learning data and used to improve the user experience.

[0577] Examples:

[0578] If the data is successfully registered in the household accounting app, the information is sent to the server and used for analyzing receipt information the next time. Also, if the user smiles with satisfaction after completing the task, the emotional information is also saved.

[0579] This embodiment allows users to efficiently handle a variety of tasks with a single OCR application, eliminating the need to use multiple dedicated applications. Furthermore, the combination of emotion engines enables optimal task suggestions based on the user's emotional state, resulting in a more intuitive and satisfying user experience.

[0580] The processing flow will be explained below.

[0581] Step 1:

[0582] The user launches the OCR application and switches to camera mode. The user presses the capture button to capture an image.

[0583] Step 2:

[0584] The captured image is temporarily stored on the device and then sent to the server.

[0585] Step 3:

[0586] The server decodes the received image and performs pre-processing such as adjusting the resolution and removing noise.

[0587] Step 4:

[0588] The server inputs the preprocessed image data into a character recognition engine, and the OCR engine extracts character information from the image.

[0589] Step 5:

[0590] The server performs spelling and grammar checks on the extracted text information and formats it into accurate text data.

[0591] Step 6:

[0592] The server inputs the text information and surrounding background information into the multi-modality artificial intelligence, and the AI ​​begins its analysis.

[0593] Step 7:

[0594] Based on the analysis results, the server's AI generates suggested tasks that are most relevant to the user and creates a task list sorted by priority.

[0595] Step 8:

[0596] The server formats the generated task list and sends it to the terminal for notification to the user.

[0597] Step 9:

[0598] The device displays the task list to the user as a popup window or alert, and the user selects one of the suggested tasks.

[0599] Step 10:

[0600] The device executes the corresponding task based on the user's selection, for example, launching a household accounting app and entering text data.

[0601] Step 11:

[0602] The device detects the user's facial expressions and tone of voice through the camera and microphone, and recognizes the user's emotions using an emotion engine.

[0603] Step 12:

[0604] The device sends the user's emotional information to the server, which then reevaluates and adjusts the task list based on the emotional information.

[0605] Step 13:

[0606] The server transmits the task list readjusted based on the emotion information to the terminal, and the terminal notifies the user again.

[0607] Step 14:

[0608] If the user selects the suggested task again, the device will execute the adjusted task, for example, prioritizing the execution of the easier task.

[0609] Step 15:

[0610] After the task is completed, the device sends the result (success, failure, feedback) to the server.

[0611] Step 16:

[0612] The server stores the results it receives in a database and uses them as training data for the AI ​​model to improve the accuracy of the next task proposal, including the emotion recognition results.

[0613] Example 2

[0614] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0615] In modern society, many users need to quickly and accurately manage various documents and payment records in their daily lives. Conventional applications using optical character recognition (OCR) technology only provide character recognition and simple data conversion, but lack the ability to manage tasks or suggest tasks based on the user's emotional state. In particular, task suggestions that take into account the user's mental and emotional state are crucial for improving the user experience. Therefore, there is a need for a system that allows users to efficiently manage documents and receive appropriate task suggestions based on their emotional state.

[0616] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0617] In this invention, the server includes means for activating a camera to acquire an image, means for preprocessing the acquired image to remove noise and optimize resolution, means for using a character recognition engine to extract character information from the preprocessed image, means for analyzing the extracted character information and surrounding background information to identify a next suggested task to be performed, means for using an emotion engine to recognize the user's emotions, means for reevaluating the task list based on the recognized emotion information, means for notifying the user of the suggested tasks, means for executing the suggested tasks based on the user's input, and means for recording the execution results and emotion information and evaluating them to improve the accuracy of the next suggestion. This enables the user to efficiently and accurately perform a series of document management and task suggestions, and further enables the user to receive optimal task suggestions adapted to their emotional state.

[0618] "Means for activating the camera" refers to a device or software that has the functionality to allow a user to activate the camera and take an image by operating an application.

[0619] "Means for preprocessing images" refers to devices or software that have the function of removing noise and optimizing resolution from acquired images, and performing processing to improve the accuracy of character recognition.

[0620] A "character recognition engine" is software or algorithms that use OCR technology to extract character information from pre-processed image data.

[0621] "Surrounding background information" refers to information other than textual information contained in an image, such as location information or user history information, and is additional data used to suggest tasks.

[0622] An "emotion engine" is software or algorithms that use a camera or microphone to analyze facial expressions and tone of voice in order to recognize a user's emotional state (e.g., joy or fatigue).

[0623] A "suggested task" is a suggestion to the user about the next action or process to be taken based on the text information and surrounding information analyzed by the system.

[0624] The "means for reevaluating the task list" refers to a function that readjusts the priority of an already generated task list based on the user's emotional information and performs the process of suggesting the most suitable task list to the user.

[0625] A "notification mechanism" is any device or software capable of displaying information in the form of a pop-up window, alert, or other form to inform the user of a proposed task.

[0626] A "means for performing a task" is any device or software that has the functionality to actually perform the proposed task selected by the user.

[0627] "Means for recording execution results" refers to devices or software that have the ability to record the results (success, failure, feedback) after a task is completed and store them in a database to improve the accuracy of suggestions next time.

[0628] An embodiment of the present invention is described below. The present invention provides a system that combines an OCR application that uses a camera to recognize characters, analyzes peripheral information, and suggests and executes the next task to be performed with an emotion engine that recognizes the user's emotions.

[0629] Camera image acquisition and preprocessing

[0630] The device activates the camera and acquires an image taken by the user. For example, the user launches an application and takes a picture of a receipt or document with the camera. This image is temporarily stored on the device and then sent to the server. The server then preprocesses the received image, removing noise and optimizing the resolution to improve the accuracy of character recognition. Specifically, it uses noise reduction algorithms such as Gaussian Blur and image resolution optimization techniques.

[0631] Example: A user takes a photo of a receipt, the image is sent from the device to a server, which removes noise from the image and scales it to the appropriate resolution.

[0632] Character Recognition Processing

[0633] The server inputs the preprocessed image data into a character recognition engine. An OCR engine (such as the open-source Tesseract or the cloud-based Google Cloud Vision API) recognizes the characters in the image and extracts them as text data. The extracted text is then checked for spelling and grammar to ensure accurate data is generated.

[0634] Example: The server extracts text information such as "product name," "price," and "store name" from a receipt image and performs spell checking and grammar correction.

[0635] Peripheral information analysis and task suggestions

[0636] The server inputs the text information extracted by the OCR engine and the surrounding background information into a multi-modality AI. The AI ​​analyzes this information and identifies the most relevant tasks for the user. As a result of the analysis, a task list is generated and sorted by priority. The generated task list is converted into a notification format and sent to the device.

[0637] Example: The server uses the extracted receipt information and background information to identify tasks such as "record this expenditure in a household accounting app" or "save a coupon" and create a list.

[0638] User emotion recognition

[0639] The device detects the user's facial expressions and tone of voice through the camera and microphone, and recognizes the user's emotions using an emotion engine. For example, it uses machine learning models for emotion recognition (Microsoft Face API or Emotion API). The recognized emotion information is sent from the device to a server.

[0640] Example: When a user takes a picture of a receipt using the smartphone camera, the emotion recognition engine recognizes the user's smile (happiness) emotion and sends that information to the server.

[0641] Emotion-based task adjustment

[0642] The server reevaluates the task list based on the emotional information received. It then adjusts the priority of tasks based on the recognized emotional information and suggests the best possible tasks for the user. For example, if the user looks tired, the list is adjusted to prioritize easier tasks.

[0643] Example: If the emotion engine detects that the user has a tired expression, the server will prioritize and suggest simplified tasks.

[0644] User notification and task execution

[0645] The device notifies the user of the task list sent from the server. The notification is displayed in the form of a pop-up window or an alert, and when the user selects one of the suggested tasks, the device executes that task. For example, it may register data in a household accounting app or launch an email sending app.

[0646] Example: When a user selects "Register in household accounting app," the device opens the household accounting app and enters receipt information.

[0647] Feedback of results

[0648] Once the task is completed, the device sends the result (success, failure, feedback) to the server. The server stores the received feedback in a database and uses it as training data for the AI ​​model to improve the accuracy of the next task suggestion. The server also stores the emotion recognition results as learning data to help improve the user experience.

[0649] Example: When data is successfully registered in a household accounting app, the information is sent to the server and used for analyzing receipt information the next time. If the user smiles with satisfaction after completing a task, the emotional information is also stored on the server.

[0650] Example prompts to input to the generative AI model

[0651] "Please explain in detail the system that analyzes receipt information and suggests appropriate tasks for the user."

[0652] "Please explain the specific steps in the process flow of a system that recognizes a user's emotions and adjusts tasks based on those emotions."

[0653] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0654] Step 1:

[0655] The device starts the camera and acquires the image taken by the user. The user starts the application and takes a picture of a receipt or document with the camera. As a result, the image data is temporarily stored in the device's local memory. The input data is the camera image, and the output data is an image file stored on the device.

[0656] Step 2:

[0657] The device sends the captured image to the server. Here, the device transfers the image data to the server using a secure communication protocol (e.g., HTTPS). The input data is the image file stored in the device, and the output data is the image file sent to the server. Specifically, data is transferred using an HTTP request.

[0658] Step 3:

[0659] The server preprocesses the images received. Preprocessing includes noise removal (for example, applying Gaussian Blur) and resolution optimization. The input data is the image file received by the server, and the output data is the preprocessed image data. Specifically, the image processing algorithm is applied using Python's OpenCV library.

[0660] Step 4:

[0661] The server inputs the preprocessed image data into a character recognition engine. An OCR engine (e.g., Tesseract) is used to recognize the characters in the image and extract them as text data. The input data is the preprocessed image data, and the output data is character string data (text). Specifically, the OCR engine is called to perform image analysis and the recognition results are obtained as text.

[0662] Step 5:

[0663] The server performs spelling and grammar checks on the extracted text data. It uses natural language processing tools (e.g., NLTK or SpaCy) to analyze the text data and format it into accurate data. The input data is text data obtained from the OCR engine, and the output data is accurately formatted text data. Specifically, it performs grammar and spelling checks using regular expressions and machine learning models.

[0664] Step 6:

[0665] The server inputs the text information extracted by the OCR engine and the surrounding background information into a multi-modality AI. The multi-modality AI analyzes this information and identifies tasks that are highly relevant to the user. The input data is precisely formatted text data and surrounding information, and the output data is a task list sorted by priority. Specifically, the data is input into the AI ​​model and task relevance is calculated.

[0666] Step 7:

[0667] The server converts the generated task list into a notification format and sends it to the terminal. The input data is the task list, and the output data is in a notification data format (e.g., JSON format). Specific operations include data formatting and sending operations.

[0668] Step 8:

[0669] The terminal notifies the user of the task list sent from the server. The notification is displayed as a pop-up window or an alert. The input data is the notification data received from the server, and the output data is the notification visually displayed to the user. Specifically, the notification is displayed using GUI (Graphical User Interface) elements.

[0670] Step 9:

[0671] The user selects one of the proposed tasks. The input data is the presented task list, and the output data is the task selected by the user. The selection is made by the user's action.

[0672] Step 10:

[0673] The device executes a task selected by the user. For example, it registers data in a household accounting app or launches an email sending app. The input data is the task selected by the user, and the output data is the task completion status (success or failure). The specific operation is to call the API of the corresponding application.

[0674] Step 11:

[0675] When the task is completed, the device sends the result (success, failure, feedback) to the server. The input data is the task completion status, and the output data is the feedback data. Specifically, the feedback is sent to the server using an HTTP request.

[0676] Step 12:

[0677] The server stores the received feedback in a database and uses it to improve the accuracy of the next task suggestion. It also stores the emotion recognition results and uses them to improve the user experience. The input data is the feedback data and the emotion recognition data, and the output data is the updated database entry. Specifically, the system stores and updates data using database queries.

[0678] (Application example 2)

[0679] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0680] In modern brick-and-mortar stores, store staff must manage multiple tasks simultaneously, requiring efficient operations. They are also required to provide high-quality service to customers, making store management more complex. Conventional systems suggest tasks without considering the user's emotional state, which can increase the burden on staff and reduce customer satisfaction. This invention provides a system that recognizes the user's emotional state and appropriately suggests and executes tasks based on that state, thereby improving the efficiency of brick-and-mortar stores and the customer experience.

[0681] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0682] In this invention, the server includes means for extracting text information from an image acquired by a camera using a character recognition engine, means for analyzing the extracted text information and surrounding background information to identify the next suggested task to be performed, means for detecting the user's facial expression and tone of voice to recognize emotions, and means for sorting the suggested tasks in order of priority based on the emotional information, thereby making it possible to suggest and execute appropriate tasks based on the emotional states of store staff and customers.

[0683] A "character recognition engine" is a device or program that analyzes and extracts character information from images acquired by a camera.

[0684] "Surrounding background information" is a general term for accompanying data and environmental information related to text information.

[0685] A "proposed task" is a proposal for the next work to be done based on the analyzed and parsed information.

[0686] "User notification means" refers to the means by which the proposed task is communicated to the user, displayed as a notification or alert.

[0687] A "task executor" is a processing mechanism for executing a suggested task based on user input.

[0688] "Facial expression detection" is a process that uses a camera to analyze a user's facial expressions and recognize their emotions.

[0689] "Tone of voice detection" is a process that uses a microphone to analyze the tone of a user's voice and recognize their emotions.

[0690] "Emotion recognition" is the process of identifying a user's emotions from facial expressions and tone of voice.

[0691] The "means for sorting by priority" is a processing function that rearranges the order of proposed tasks based on their importance and ease of execution.

[0692] "Multi-modality artificial intelligence" is an artificial intelligence system that combines and analyzes multiple data formats (for example, text information and background information) to provide users with appropriate information.

[0693] This invention uses a "smart store assistant" system in a physical store to efficiently and effectively suggest tasks to store staff and customers and assist them in carrying them out. A specific embodiment of this system will be described below.

[0694] First, the device uses a camera to scan the barcode or label of a product in a physical store. This device can be a smartphone or smart glasses. The camera is activated and the captured image is temporarily stored on the device. The image is then sent to a server for pre-processing. This pre-processing involves removing noise from the image and adjusting its resolution to improve the accuracy of character recognition by the OCR engine.

[0695] Next, the server inputs the preprocessed image data into an OCR engine, such as Tesseract OCR or Google Cloud Vision API. This extracts the text information from the image and records it as text data. At this stage, information such as the product name, price, and expiration date are obtained.

[0696] The extracted text information is input into a multi-modality AI system. The server analyzes the text information in combination with the surrounding contextual information to identify tasks to be proposed. For example, this could include suggesting restocking of low-stock items or proposing promotions for specific products.

[0697] The device detects the user's facial expressions and tone of voice through a camera and microphone, and analyzes them with an emotion recognition engine. The recognized emotion information is sent to the server and reflected in task suggestions. For example, if the user is tired, the device will prioritize simple tasks to reduce the user's burden.

[0698] Finally, the server sends the prioritized task list to the device. The device notifies the user of the task list through a pop-up window or an alert. When the user selects one of the suggested tasks, the task is automatically executed. For example, it may send a stock replenishment request or start a sales promotion campaign. Once the task is completed, the device feeds back the results to the server. The server stores this in a database and uses it to improve the accuracy of future suggestions.

[0699] For example, here's a prompt to notify you when an item is low in stock:

[0700] Scan the receipt image.

[0701] Important Task: This item is low on stock and needs to be restocked. Would you like to submit a restock request?

[0702] This system will reduce the burden on store staff in physical stores, enable them to carry out their work more efficiently, and improve customer satisfaction.

[0703] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0704] Step 1:

[0705] The device uses a camera to scan products or labels in a physical store. The user picks up the device (smartphone or smart glasses) and points the camera at the target product to capture an image. In this step, the camera is activated and the captured image is temporarily saved in the device. The image is then sent to the server. The input is the image of the product or label, and the output is the image data sent to the server.

[0706] Step 2:

[0707] The server preprocesses the received image data. Preprocessing includes noise removal and resolution adjustment. Specifically, the server converts the image to grayscale and applies Gaussian blur to remove noise. It also adjusts the resolution appropriately to improve the accuracy of character recognition. The input is the received raw image data, and the output is preprocessed, high-quality image data.

[0708] Step 3:

[0709] The server inputs the preprocessed image data into the OCR engine. The OCR engine (for example, Tesseract OCR or Google Cloud Vision API) analyzes the character information in the image and extracts it as text data. In this step, the server calls the OCR engine to perform the analysis and saves the obtained text data on the server. The input is the preprocessed image data, and the output is text data containing character information.

[0710] Step 4:

[0711] The server inputs the extracted text information and surrounding background information into a multi-modality AI for analysis. This identifies suggested tasks that are highly relevant to the user. The server then integrates the text information and background information and analyzes it using an AI model. As a result, a list of suggested tasks is generated. The input is text information and background information, and the output is a list of suggested tasks.

[0712] Step 5:

[0713] The device detects the user's facial expressions and tone of voice through a camera and microphone, and analyzes them using an emotion recognition engine. When the user operates the device, the camera captures the user's facial expressions and the microphone records the user's tone of voice. This data is sent from the device to a server and analyzed by the emotion recognition engine. The input is facial expression and voice data, and the output is recognized emotional information.

[0714] Step 6:

[0715] The server re-evaluates the proposed task list based on the emotion information and sorts it in order of priority. The server receives the emotion recognition results and adjusts the task priority according to the user's state. For example, if the user is tired, it prioritizes easier tasks. The input is the proposed task list and emotion information, and the output is a task list sorted in order of priority.

[0716] Step 7:

[0717] The terminal notifies the user of a priority-ordered task list sent from the server. The terminal displays the task list in a pop-up window or alert format, allowing the user to select from the list. When the user selects a specific task, that task is executed. The input is a task list sorted by priority, and the output is a notification to the user and the execution of the task.

[0718] Step 8:

[0719] After completing a task, the device sends the result (success, failure, feedback) to the server. The server stores the received feedback in a database and uses it as training data for the AI ​​model to improve the accuracy of the next task proposal. The input is the task execution result and feedback, and the output is the feedback information stored as learning data.

[0720] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0721] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0722] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0723] [Third embodiment]

[0724] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0725] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0726] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0727] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0728] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0729] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0730] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0731] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0732] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0733] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0734] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0735] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0736] An embodiment of the present invention will be described below. The present invention provides an OCR application that uses a camera to recognize characters, analyzes surrounding information, and suggests and executes the next task to be performed.

[0737] 1. Camera image acquisition and preprocessing

[0738] The device activates the camera and captures the image taken by the user. For example, the user activates a scanner application and takes a picture of a receipt or document with the camera. The image is temporarily stored on the device and then sent to the server. The server preprocesses the received image, removing noise and optimizing the resolution to improve the accuracy of character recognition.

[0739] Examples:

[0740] The user takes a photo of the receipt, and the image is sent from the device to a server where noise is removed and resolution is adjusted.

[0741] 2. Character Recognition Processing

[0742] The server inputs the preprocessed image data into a character recognition engine. The OCR engine (e.g., an open-source OCR tool or a cloud-based OCR API) recognizes the characters in the image and extracts them as text data. The extracted text is then checked for spelling and grammar before being formatted into accurate data.

[0743] Examples:

[0744] The server extracts text information such as the product name, price, and store name from the receipt image.

[0745] 3. Analysis of surrounding information and task proposal

[0746] The server inputs the text information extracted by the OCR engine and the surrounding background information into a multi-modality AI. The AI ​​analyzes this information and identifies the most relevant tasks for the user. As a result of the analysis, a task list is generated and sorted by priority. This task list is then converted into a format that can be displayed to the user and sent to the device.

[0747] Examples:

[0748] The server uses the extracted receipt information and background information to suggest tasks such as "register this expenditure in a household accounting app" or "save a coupon."

[0749] 4. User Notification and Task Execution

[0750] The device notifies the user of the task list sent from the server. The notification is displayed in the form of a pop-up window or an alert. When the user selects one of the suggested tasks, the device executes that task. For example, it may register data in a household accounting app or launch an email sending app.

[0751] Examples:

[0752] When the user selects "Register in household accounting app," the device opens the household accounting app and enters the receipt information.

[0753] 5. Feedback of results

[0754] Once the task is completed, the device sends the result (success, failure, feedback) to the server, which stores the received feedback in a database and uses it as training data for the AI ​​model to improve the accuracy of the next task suggestions.

[0755] Examples:

[0756] If the data is successfully registered in the household accounting app, the information is sent to the server and used for analyzing receipt information the next time.

[0757] This embodiment allows users to efficiently handle various tasks with one OCR application, eliminating the need to use multiple dedicated applications. Furthermore, the use of multi-modality artificial intelligence improves the accuracy of suggested tasks and makes user operation easier and more intuitive.

[0758] The processing flow will be explained below.

[0759] Step 1:

[0760] The user launches the OCR application and switches to camera mode. The user presses the capture button to capture an image.

[0761] Step 2:

[0762] The captured image is temporarily stored on the device and then sent to the server.

[0763] Step 3:

[0764] The server decodes the received image into a file format and performs preprocessing such as adjusting the resolution and removing noise.

[0765] Step 4:

[0766] The server inputs the preprocessed image data into a character recognition engine, and the OCR engine extracts character information from the image.

[0767] Step 5:

[0768] The server performs spelling and grammar checks on the extracted text information and formats it into accurate text data.

[0769] Step 6:

[0770] The server inputs the text information and surrounding background information into the multi-modality artificial intelligence, and the AI ​​begins its analysis.

[0771] Step 7:

[0772] Based on the analysis results, the server's AI generates suggested tasks that are most relevant to the user and creates a task list sorted by priority.

[0773] Step 8:

[0774] The server formats the generated task list and sends it to the terminal for notification to the user.

[0775] Step 9:

[0776] The device displays the task list to the user as a popup window or alert, and the user selects one of the suggested tasks.

[0777] Step 10:

[0778] The device executes the corresponding task based on the user's selection, for example, launching a household accounting app and entering text data.

[0779] Step 11:

[0780] The server receives the task execution results (success, failure, feedback) from the terminal and stores them in a database.

[0781] Step 12:

[0782] The server uses the saved result data as learning data for the AI ​​and updates the model to improve the accuracy of the next proposed task.

[0783] Example 1

[0784] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0785] Conventional OCR applications have insufficient character recognition accuracy, often requiring users to manually correct information. Furthermore, information analysis and task suggestions after character recognition are ineffective, resulting in poor user convenience. Furthermore, the accuracy of the suggested tasks is low, and there is a lack of a learning mechanism that utilizes feedback.

[0786] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0787] In this invention, the server includes means for temporarily saving images acquired from the camera in an internal storage device, means for transmitting images from the internal storage device to the server and performing noise reduction and resolution optimization, means for using a character recognition engine to extract character information from the noise-removed and resolution-optimized images, means for analyzing the extracted character information and surrounding background information using multi-modality artificial intelligence and identifying a suggested next task to be performed, means for notifying the user of the suggested task, means for executing the suggested task based on user input, and means for sending feedback after task execution to the server and storing the feedback in a database to use as training data. This enables improved character recognition accuracy, effective information analysis and task suggestions, and a learning mechanism using feedback.

[0788] A "camera" is a device for taking still images and videos.

[0789] "Internal storage" means a storage medium within a device that is used to store data temporarily or permanently.

[0790] A "server" is a computer system that provides services to other computers on a network.

[0791] "Noise reduction" is the process of removing unwanted interference and noise from image and audio data.

[0792] "Resolution optimization" is a process of adjusting the image so that the details can be displayed clearly.

[0793] A "character recognition engine" is software or an algorithm that identifies characters in an image and extracts them as text data.

[0794] "Multi-modality artificial intelligence" is artificial intelligence that integrates, understands, and analyzes data in multiple different formats (e.g., text and images).

[0795] "Analysis" is the process of examining data in detail to understand its meaning and structure.

[0796] A "suggested task" is a specific action or task that is shown to the user to take next.

[0797] "Notification means" refers to a method or device for notifying a user of information.

[0798] "Feedback" refers to reporting the results of a task and user evaluation information to the server, and is used for future improvements.

[0799] A "database" is a system for efficiently storing, managing, and searching data.

[0800] An embodiment of the present invention will be described in detail. The invention provides an OCR application that uses a camera to recognize characters, analyzes surrounding information, and proposes and executes the next task to be performed. The following describes the flow of data processing and data calculation using each hardware and software.

[0801] 1. Camera image acquisition and preprocessing

[0802] The device activates the camera and acquires the image taken by the user. For example, the user activates a scanner application and takes a picture of a receipt or document with the camera. This image is temporarily stored in the device's internal storage and then sent to the server. The server preprocesses the received image, removing noise and optimizing the resolution to improve the accuracy of character recognition. For example, it uses the image processing library "OpenCV" to remove noise and optimizes the resolution using the "PhotoShop API."

[0803] Examples:

[0804] The user takes a photo of the receipt, and the image is sent from the device to a server where noise is removed and resolution is adjusted.

[0805] 2. Character Recognition Processing

[0806] The server inputs the preprocessed image data into a character recognition engine. The OCR engine used can be the open source "Tesseract" or the cloud-based "Google Cloud Vision API." The server calls the OCR engine, recognizes the characters in the image, and extracts them as text data. The extracted text is then checked for spelling and grammar using a natural language processing tool (such as "NLTK" or "SpaCy"), and formatted into accurate text data.

[0807] Examples:

[0808] The server extracts text information such as the product name, price, and store name from the receipt image.

[0809] 3. Analysis of surrounding information and task proposal

[0810] The server inputs the text information extracted by the OCR engine and the surrounding background information into a multi-modality AI (such as GPT-3 or BERT). The AI ​​analyzes this information and identifies the most relevant tasks for the user. As a result of the analysis, a task list is generated and sorted by priority. This task list is then converted into a format that can be displayed to the user and sent to the device.

[0811] Examples:

[0812] The server uses the extracted receipt information and background information to suggest tasks such as "register this expenditure in a household accounting app" or "save a coupon."

[0813] 4. User Notification and Task Execution

[0814] The device notifies the user of the task list sent from the server. The notification is displayed in the form of a pop-up window or an alert. When the user selects one of the suggested tasks, the device executes that task. Specifically, it registers data in a household accounting app or launches an email sending app.

[0815] Examples:

[0816] When the user selects "Register in household accounting app," the device opens the household accounting app and inputs the receipt information.

[0817] 5. Feedback of results

[0818] Once the task is completed, the device sends the result (success, failure, feedback) to the server, which stores the received feedback in a database and uses it as training data for the AI ​​model to improve the accuracy of the next task suggestions.

[0819] Examples:

[0820] If the data is successfully registered in the household accounting app, the information is sent to the server and used for analyzing receipt information the next time.

[0821] Prompt Sentence Examples

[0822] In a system for implementing the invention, more specific task suggestions can be made by using prompt sentences such as the following to input to a generative AI model:

[0823] example:

[0824] "I took a photo of my receipt. Can you give me some suggestions to help me manage my household finances and coupons?"

[0825] The above is an embodiment of the present invention. This embodiment allows users to efficiently handle a variety of tasks with a single OCR application, eliminating the need to use multiple dedicated applications. Furthermore, by utilizing multi-modality artificial intelligence, the accuracy of the suggested tasks is improved, and user operation becomes simpler and more intuitive.

[0826] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0827] Step 1:

[0828] A user starts a scanner application on their device and uses the camera to take a picture of a receipt or document. At this time, the device's camera captures the image data and temporarily stores it in the device's internal storage. The input is the receipt image taken by the user, and the output is the image data stored in the internal storage.

[0829] Step 2:

[0830] The device sends image data stored in its internal storage device to the server. The image data is sent over the Internet using a secure communication protocol such as HTTPS. The input is the image data in the internal storage device, and the output is the image data sent to the server.

[0831] Step 3:

[0832] The server analyzes the image data it receives and performs preprocessing to remove noise and optimize resolution. Specifically, it removes noise using the image processing library "OpenCV" and optimizes resolution using "PhotoShop API." The input is the image data sent to the server, and the output is image data with noise removed and resolution optimized.

[0833] Step 4:

[0834] The server inputs the preprocessed image data into a character recognition engine. The OCR engine used is "Tesseract" or "Google Cloud Vision API." The server uses the OCR engine to recognize characters in the image and extract them as text data. The input is noise-removed and resolution-optimized image data, and the output is the extracted text data.

[0835] Step 5:

[0836] The server uses natural language processing tools (such as NLTK and SpaCy) to check the spelling and grammar of the extracted text data, which then converts it into accurate text data. The input is text data obtained from the OCR engine, and the output is accurate text data that has been checked for spelling and grammar.

[0837] Step 6:

[0838] The server inputs the text data extracted by the OCR engine and the surrounding context into a multi-modality AI model (e.g., GPT-3 or BERT). The AI ​​model analyzes this information and identifies tasks that are highly relevant to the user. The input is the formatted text data and context, and the output is a list of suggested tasks.

[0839] Step 7:

[0840] The server converts the generated task list into a notification format and sends it to the terminal. For example, it constructs the task list in JSON format and sends it to the terminal via the Internet. The input is the specified task list, and the output is the task list sent to the terminal.

[0841] Step 8:

[0842] The terminal analyzes the task list received from the server and notifies the user. The notification is displayed in the form of a pop-up window or an alert. The input is the task list sent from the server, and the output is the tasks to be notified to the user.

[0843] Step 9:

[0844] The user selects one of the proposed tasks, for example, registering data in a household accounting app. The input is the notified task list, and the output is the specific task selected.

[0845] Step 10:

[0846] The device executes the selected task. Specifically, it opens a household accounting app and automatically inputs the receipt information the user wants to register. The input is the selected task information, and the output is the executed task.

[0847] Step 11:

[0848] The terminal sends the result (success, failure, feedback) after completing the task to the server. The result data is sent to the server via the Internet. The input is the result of the task execution, and the output is the feedback data sent to the server.

[0849] Step 12:

[0850] The server stores the received feedback in a database. The feedback data is used as training data for the AI ​​model to improve the accuracy of the next task suggestion. The input is the feedback data sent from the device, and the output is the updated database information.

[0851] (Application example 1)

[0852] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0853] In physical stores, there is a need for support to enable users to quickly and accurately obtain product information and choose appropriate actions based on that information. However, current systems require manual information collection and analysis, which is extremely time-consuming. In addition, there is a lack of systems that allow users to easily compare prices and product reviews at other stores while shopping in a store. Therefore, technology is needed to enable users to shop efficiently and effectively.

[0854] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0855] In this invention, the server includes means for extracting text information from an image acquired by a camera using a character recognition engine, means for analyzing the extracted text information and surrounding background information and identifying a suggested next task to be performed, means for notifying the user of the suggested task, means for executing the suggested task based on the user's input, means for performing preprocessing such as image noise reduction and resolution optimization, means for performing cloud-based analysis, and means for suggesting tasks such as comparing product prices and displaying product reviews. This enables users to make optimal shopping decisions in real time based on product and shelf information captured by the camera.

[0856] A "character recognition engine" is software or hardware that recognizes characters from images acquired from a camera or other imaging device and extracts them as digital text data.

[0857] "Preprocessing" refers to processing of the captured image, such as noise removal and resolution optimization, to improve the accuracy of character recognition.

[0858] "Cloud-based analysis" is a technology that uses computing resources in a cloud environment to analyze and calculate data, enabling high-speed, large-scale data processing.

[0859] "Multi-modality AI" is an AI technology that integrates multiple different data modalities (e.g., text, images, audio), analyzes them comprehensively, and makes highly accurate task suggestions and predictions.

[0860] A "suggested task" is the next action or task that the AI ​​identifies based on user input and acquired data.

[0861] "Noise reduction" is a process for removing unnecessary information and data in image preprocessing to improve image quality.

[0862] "Resolution optimization" is a process that optimizes the fine details of an image to improve the accuracy of character recognition.

[0863] "User notification" means the notification method used by the system to inform the user of required information or tasks, including pop-up windows and alerts.

[0864] "Task execution" means actually performing the proposed task selected by the user, which includes launching an application and registering data.

[0865] "Price comparison" refers to the process of collecting and comparing price information for a specific product at different sales locations and times to provide users with the best purchasing options.

[0866] "Review display" refers to displaying ratings and impressions posted by other users about a specific product, allowing users to refer to other people's opinions before purchasing.

[0867] The system for implementing the present invention includes the following detailed processes to support users' shopping activities.

[0868] The programs implemented on the server and terminal first include a means for acquiring and preprocessing camera images. The terminal's camera is activated, and the user takes a photo of the product or shelf information. The captured image is temporarily saved on the terminal and then sent to the server. The server uses the image processing library OpenCV (cv2) to remove noise and optimize the resolution of the received image data. After preprocessing, the image is input into the character recognition engine Tesseract (pytesseract), and character information such as the product name, price, and features is extracted as text data.

[0869] The extracted text data and surrounding contextual information are analyzed by a multi-modality AI on the server to identify new tasks. This AI model performs cloud-based analysis using the Google Cloud Vision API to generate optimal suggested tasks for the user (e.g., comparing product prices or displaying reviews) and sort them by priority. The resulting task list is notified to the user and sent to the device. Notifications are displayed as pop-up windows or alerts on the device.

[0870] The user checks the notification and selects one of the suggested tasks. For example, if they select the task of comparing prices of a specific product with other stores, the device will launch a web browser and display the relevant search page. For other tasks, such as displaying product reviews, the device will display review information from appropriate sites. This process allows the user to efficiently proceed with their shopping.

[0871] Furthermore, the results of the task execution are sent to the server as feedback and saved as learning data to improve the accuracy of the next task proposal. The server uses this feedback to continuously improve the accuracy of the generative AI model.

[0872] Specific prompt examples:

[0873] Prompt after taking a photo of the receipt:

[0874] Your receipt information has been extracted. What do you want to do with this information? Choose from the following options:

[0875] 1. Register for a household accounting app

[0876] 2. Save the coupon

[0877] 3. Compare prices with other stores

[0878] Product comparison prompt:

[0879] Enter the name of the product you want to search for:

[0880] The system allows users to efficiently gather product information and compare prices based on captured images, enabling them to make the best purchasing decision. Specific hardware components include smartphones and smart glasses, while software components include OpenCV, Tesseract, Google Cloud Vision API, and multi-modality artificial intelligence.

[0881] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0882] Step 1: Acquire camera images

[0883] The device activates the camera, and the user takes a picture of the product or shelf information. The input is image data acquired from the device's camera, and the output is an image file temporarily stored in the device. This image file is used in the next processing step.

[0884] Step 2: Image preprocessing

[0885] The device sends the saved image file to the server. The server performs noise removal and resolution optimization on the received image data. The input is the image data sent from the device, and the output is a preprocessed image that has been noise removed and resolution optimized. This process uses an image processing library called OpenCV.

[0886] Step 3: Character Recognition

[0887] The server inputs the preprocessed image data into the OCR engine (Tesseract). The OCR engine recognizes the characters in the image and extracts them as text data. The input is the preprocessed image data, and the output is the extracted text data. Specifically, it detects the outlines of the characters in the image and recognizes them as characters.

[0888] Step 4: Analyze text and context

[0889] The server inputs the text information extracted by the OCR engine and the surrounding background information into a multi-modality artificial intelligence. The AI ​​analyzes this information and suggests related tasks to the user (for example, comparing product prices or displaying reviews). The input is the extracted text data and background information, and the output is a task list as the analysis result. Specifically, it extracts product features from the text data and generates related tasks based on this.

[0890] Step 5: Notify users

[0891] The server sends the generated task list to the terminal and notifies the user. The notification is displayed on the terminal as a pop-up window or alert. The input is the task list as the analysis result, and the output is the notification to the user. Specifically, it lists tasks in order of priority based on specific conditions and presents this to the user.

[0892] Step 6: User Task Selection

[0893] The user selects one of the suggested tasks. The input is the task list presented as a notification, and the output is the user's selection. The specific behavior involves selecting a task through a user interface.

[0894] Step 7: Execute the task

[0895] The device executes the selected task based on the user's input. For example, if a user wants to compare the prices of a particular product with other stores, the device will launch a web browser and display the relevant page. If a user wants to view product reviews, the device will display the review information from the appropriate website. The input is the user's selection, and the output is the result of each task. Specific operations include launching an application or displaying data according to the selected task.

[0896] Step 8: Feedback on results

[0897] The results of the task execution are sent from the device to the server. The server stores this feedback in a database and uses it as training data to improve the accuracy of the next task suggestion. The input is the task execution result, and the output is a database entry for the next training. Specific operations include evaluating the execution results and recording them.

[0898] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0899] An embodiment of the present invention is described below. The present invention provides a system that combines an OCR application that uses a camera to recognize characters, analyzes peripheral information, and suggests and executes the next task to be performed with an emotion engine that recognizes the user's emotions.

[0900] 1. Camera image acquisition and preprocessing

[0901] The device activates the camera and captures the image taken by the user. For example, the user launches an application and takes a picture of a receipt or document with the camera. The image is temporarily stored on the device and then sent to the server. The server preprocesses the received image, removing noise and optimizing the resolution to improve the accuracy of character recognition.

[0902] Examples:

[0903] The user takes a photo of the receipt, and the image is sent from the device to a server where noise is removed and resolution is adjusted.

[0904] 2. Character Recognition Processing

[0905] The server inputs the preprocessed image data into a character recognition engine. The OCR engine (e.g., an open-source OCR tool or a cloud-based OCR API) recognizes the characters in the image and extracts them as text data. The extracted text is then checked for spelling and grammar before being formatted into accurate data.

[0906] Examples:

[0907] The server extracts text information such as the product name, price, and store name from the receipt image.

[0908] 3. Analysis of surrounding information and task proposal

[0909] The server inputs the text information extracted by the OCR engine and the surrounding background information into a multi-modality AI. The AI ​​analyzes this information and identifies the most relevant tasks for the user. As a result of the analysis, a task list is generated and sorted by priority. This task list is then converted into a format that can be displayed to the user and sent to the device.

[0910] Examples:

[0911] The server uses the extracted receipt information and background information to suggest tasks such as "register this expenditure in a household accounting app" or "save a coupon."

[0912] 4. User Emotion Recognition

[0913] The device detects the user's facial expressions and tone of voice through the camera and microphone, and recognizes the user's emotions using an emotion engine. For example, when a user takes a photo of a receipt, the camera detects the user's smile. This information is sent from the device to the server.

[0914] Examples:

[0915] When a user takes a photo of a receipt using their smartphone camera, the emotion engine recognizes the user's smile (happiness) emotion.

[0916] 5. Emotion-Based Task Adjustment

[0917] The server re-evaluates the task list based on the emotional information recognized by the emotion engine and prioritizes tasks that suit the user's emotions. For example, if the user looks tired, it will prioritize easy tasks.

[0918] Examples:

[0919] The emotion engine detects when the user has a tired expression, and the server prioritizes and suggests a simplified task such as "immediately record this expenditure in a household accounting app."

[0920] 6. User Notification and Task Execution

[0921] The device notifies the user of the task list sent from the server. The notification is displayed in the form of a pop-up window or an alert. When the user selects one of the suggested tasks, the device executes that task. For example, it may register data in a household accounting app or launch an email sending app.

[0922] Examples:

[0923] When the user selects "Register in household accounting app," the device opens the household accounting app and enters the receipt information.

[0924] 7. Feedback of results

[0925] Once the task is completed, the device sends the result (success, failure, feedback) to the server. The server stores the received feedback in a database and uses it as training data for the AI ​​model to improve the accuracy of the next task suggestions. Emotion recognition results are also stored as learning data and used to improve the user experience.

[0926] Examples:

[0927] If the data is successfully registered in the household accounting app, the information is sent to the server and used for analyzing receipt information the next time. Also, if the user smiles with satisfaction after completing the task, the emotional information is also saved.

[0928] This embodiment allows users to efficiently handle a variety of tasks with a single OCR application, eliminating the need to use multiple dedicated applications. Furthermore, the combination of emotion engines enables optimal task suggestions based on the user's emotional state, resulting in a more intuitive and satisfying user experience.

[0929] The processing flow will be explained below.

[0930] Step 1:

[0931] The user launches the OCR application and switches to camera mode. The user presses the capture button to capture an image.

[0932] Step 2:

[0933] The captured image is temporarily stored on the device and then sent to the server.

[0934] Step 3:

[0935] The server decodes the received image and performs pre-processing such as adjusting the resolution and removing noise.

[0936] Step 4:

[0937] The server inputs the preprocessed image data into a character recognition engine, and the OCR engine extracts character information from the image.

[0938] Step 5:

[0939] The server performs spelling and grammar checks on the extracted text information and formats it into accurate text data.

[0940] Step 6:

[0941] The server inputs the text information and surrounding background information into the multi-modality artificial intelligence, and the AI ​​begins its analysis.

[0942] Step 7:

[0943] Based on the analysis results, the server's AI generates suggested tasks that are most relevant to the user and creates a task list sorted by priority.

[0944] Step 8:

[0945] The server formats the generated task list and sends it to the terminal for notification to the user.

[0946] Step 9:

[0947] The device displays the task list to the user as a popup window or alert, and the user selects one of the suggested tasks.

[0948] Step 10:

[0949] The device executes the corresponding task based on the user's selection, for example, launching a household accounting app and entering text data.

[0950] Step 11:

[0951] The device detects the user's facial expressions and tone of voice through the camera and microphone, and recognizes the user's emotions using an emotion engine.

[0952] Step 12:

[0953] The device sends the user's emotional information to the server, which then reevaluates and adjusts the task list based on the emotional information.

[0954] Step 13:

[0955] The server transmits the task list readjusted based on the emotion information to the terminal, and the terminal notifies the user again.

[0956] Step 14:

[0957] If the user selects the suggested task again, the device will execute the adjusted task, for example, prioritizing the execution of the easier task.

[0958] Step 15:

[0959] After the task is completed, the device sends the result (success, failure, feedback) to the server.

[0960] Step 16:

[0961] The server stores the results it receives in a database and uses them as training data for the AI ​​model to improve the accuracy of the next task proposal, including the emotion recognition results.

[0962] Example 2

[0963] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0964] In modern society, many users need to quickly and accurately manage various documents and payment records in their daily lives. Conventional applications using optical character recognition (OCR) technology only provide character recognition and simple data conversion, but lack the ability to manage tasks or suggest tasks based on the user's emotional state. In particular, task suggestions that take into account the user's mental and emotional state are crucial for improving the user experience. Therefore, there is a need for a system that allows users to efficiently manage documents and receive appropriate task suggestions based on their emotional state.

[0965] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0966] In this invention, the server includes means for activating a camera to acquire an image, means for preprocessing the acquired image to remove noise and optimize resolution, means for using a character recognition engine to extract character information from the preprocessed image, means for analyzing the extracted character information and surrounding background information to identify a next suggested task to be performed, means for using an emotion engine to recognize the user's emotions, means for reevaluating the task list based on the recognized emotion information, means for notifying the user of the suggested tasks, means for executing the suggested tasks based on the user's input, and means for recording the execution results and emotion information and evaluating them to improve the accuracy of the next suggestion. This enables the user to efficiently and accurately perform a series of document management and task suggestions, and further enables the user to receive optimal task suggestions adapted to their emotional state.

[0967] "Means for activating the camera" refers to a device or software that has the functionality to allow a user to activate the camera and take an image by operating an application.

[0968] "Means for preprocessing images" refers to devices or software that have the function of removing noise and optimizing resolution from acquired images, and performing processing to improve the accuracy of character recognition.

[0969] A "character recognition engine" is software or algorithms that use OCR technology to extract character information from pre-processed image data.

[0970] "Surrounding background information" refers to information other than textual information contained in an image, such as location information or user history information, and is additional data used to suggest tasks.

[0971] An "emotion engine" is software or algorithms that use a camera or microphone to analyze facial expressions and tone of voice in order to recognize a user's emotional state (e.g., joy or fatigue).

[0972] A "suggested task" is a suggestion to the user about the next action or process to be taken based on the text information and surrounding information analyzed by the system.

[0973] The "means for reevaluating the task list" refers to a function that readjusts the priority of an already generated task list based on the user's emotional information and performs the process of suggesting the most suitable task list to the user.

[0974] A "notification mechanism" is any device or software capable of displaying information in the form of a pop-up window, alert, or other form to inform the user of a proposed task.

[0975] A "means for performing a task" is any device or software that has the functionality to actually perform the proposed task selected by the user.

[0976] "Means for recording execution results" refers to devices or software that have the ability to record the results (success, failure, feedback) after a task is completed and store them in a database to improve the accuracy of suggestions next time.

[0977] An embodiment of the present invention is described below. The present invention provides a system that combines an OCR application that uses a camera to recognize characters, analyzes peripheral information, and suggests and executes the next task to be performed with an emotion engine that recognizes the user's emotions.

[0978] Camera image acquisition and preprocessing

[0979] The device activates the camera and acquires an image taken by the user. For example, the user launches an application and takes a picture of a receipt or document with the camera. This image is temporarily stored on the device and then sent to the server. The server then preprocesses the received image, removing noise and optimizing the resolution to improve the accuracy of character recognition. Specifically, it uses noise reduction algorithms such as Gaussian Blur and image resolution optimization techniques.

[0980] Example: A user takes a photo of a receipt, the image is sent from the device to a server, which removes noise from the image and scales it to the appropriate resolution.

[0981] Character Recognition Processing

[0982] The server inputs the preprocessed image data into a character recognition engine. An OCR engine (such as the open-source Tesseract or the cloud-based Google Cloud Vision API) recognizes the characters in the image and extracts them as text data. The extracted text is then checked for spelling and grammar to ensure accurate data is generated.

[0983] Example: The server extracts text information such as "product name," "price," and "store name" from a receipt image and performs spell checking and grammar correction.

[0984] Peripheral information analysis and task suggestions

[0985] The server inputs the text information extracted by the OCR engine and the surrounding background information into a multi-modality AI. The AI ​​analyzes this information and identifies the most relevant tasks for the user. As a result of the analysis, a task list is generated and sorted by priority. The generated task list is converted into a notification format and sent to the device.

[0986] Example: The server uses the extracted receipt information and background information to identify tasks such as "record this expenditure in a household accounting app" or "save a coupon" and create a list.

[0987] User emotion recognition

[0988] The device detects the user's facial expressions and tone of voice through the camera and microphone, and recognizes the user's emotions using an emotion engine. For example, it uses machine learning models for emotion recognition (Microsoft Face API or Emotion API). The recognized emotion information is sent from the device to a server.

[0989] Example: When a user takes a picture of a receipt using the smartphone camera, the emotion recognition engine recognizes the user's smile (happiness) emotion and sends that information to the server.

[0990] Emotion-based task adjustment

[0991] The server reevaluates the task list based on the emotional information received. It then adjusts the priority of tasks based on the recognized emotional information and suggests the best possible tasks for the user. For example, if the user looks tired, the list is adjusted to prioritize easier tasks.

[0992] Example: If the emotion engine detects that the user has a tired expression, the server will prioritize and suggest simplified tasks.

[0993] User notification and task execution

[0994] The device notifies the user of the task list sent from the server. The notification is displayed in the form of a pop-up window or an alert, and when the user selects one of the suggested tasks, the device executes that task. For example, it may register data in a household accounting app or launch an email sending app.

[0995] Example: When a user selects "Register in household accounting app," the device opens the household accounting app and enters receipt information.

[0996] Feedback of results

[0997] Once the task is completed, the device sends the result (success, failure, feedback) to the server. The server stores the received feedback in a database and uses it as training data for the AI ​​model to improve the accuracy of the next task suggestion. The server also stores the emotion recognition results as learning data to help improve the user experience.

[0998] Example: When data is successfully registered in a household accounting app, the information is sent to the server and used for analyzing receipt information the next time. If the user smiles with satisfaction after completing a task, the emotional information is also stored on the server.

[0999] Example prompts to input to the generative AI model

[1000] "Please explain in detail the system that analyzes receipt information and suggests appropriate tasks for the user."

[1001] "Please explain the specific steps in the process flow of a system that recognizes a user's emotions and adjusts tasks based on those emotions."

[1002] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1003] Step 1:

[1004] The device starts the camera and acquires the image taken by the user. The user starts the application and takes a picture of a receipt or document with the camera. As a result, the image data is temporarily stored in the device's local memory. The input data is the camera image, and the output data is an image file stored on the device.

[1005] Step 2:

[1006] The device sends the captured image to the server. Here, the device transfers the image data to the server using a secure communication protocol (e.g., HTTPS). The input data is the image file stored in the device, and the output data is the image file sent to the server. Specifically, data is transferred using an HTTP request.

[1007] Step 3:

[1008] The server preprocesses the images received. Preprocessing includes noise removal (for example, applying Gaussian Blur) and resolution optimization. The input data is the image file received by the server, and the output data is the preprocessed image data. Specifically, the image processing algorithm is applied using Python's OpenCV library.

[1009] Step 4:

[1010] The server inputs the preprocessed image data into a character recognition engine. An OCR engine (e.g., Tesseract) is used to recognize the characters in the image and extract them as text data. The input data is the preprocessed image data, and the output data is character string data (text). Specifically, the OCR engine is called to perform image analysis and the recognition results are obtained as text.

[1011] Step 5:

[1012] The server performs spelling and grammar checks on the extracted text data. It uses natural language processing tools (e.g., NLTK or SpaCy) to analyze the text data and format it into accurate data. The input data is text data obtained from the OCR engine, and the output data is accurately formatted text data. Specifically, it performs grammar and spelling checks using regular expressions and machine learning models.

[1013] Step 6:

[1014] The server inputs the text information extracted by the OCR engine and the surrounding background information into a multi-modality AI. The multi-modality AI analyzes this information and identifies tasks that are highly relevant to the user. The input data is precisely formatted text data and surrounding information, and the output data is a task list sorted by priority. Specifically, the data is input into the AI ​​model and task relevance is calculated.

[1015] Step 7:

[1016] The server converts the generated task list into a notification format and sends it to the terminal. The input data is the task list, and the output data is in a notification data format (e.g., JSON format). Specific operations include data formatting and sending operations.

[1017] Step 8:

[1018] The terminal notifies the user of the task list sent from the server. The notification is displayed as a pop-up window or an alert. The input data is the notification data received from the server, and the output data is the notification visually displayed to the user. Specifically, the notification is displayed using GUI (Graphical User Interface) elements.

[1019] Step 9:

[1020] The user selects one of the proposed tasks. The input data is the presented task list, and the output data is the task selected by the user. The selection is made by the user's action.

[1021] Step 10:

[1022] The device executes a task selected by the user. For example, it registers data in a household accounting app or launches an email sending app. The input data is the task selected by the user, and the output data is the task completion status (success or failure). The specific operation is to call the API of the corresponding application.

[1023] Step 11:

[1024] When the task is completed, the device sends the result (success, failure, feedback) to the server. The input data is the task completion status, and the output data is the feedback data. Specifically, the feedback is sent to the server using an HTTP request.

[1025] Step 12:

[1026] The server stores the received feedback in a database and uses it to improve the accuracy of the next task suggestion. It also stores the emotion recognition results and uses them to improve the user experience. The input data is the feedback data and the emotion recognition data, and the output data is the updated database entry. Specifically, the system stores and updates data using database queries.

[1027] (Application example 2)

[1028] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1029] In modern brick-and-mortar stores, store staff must manage multiple tasks simultaneously, requiring efficient operations. They are also required to provide high-quality service to customers, making store management more complex. Conventional systems suggest tasks without considering the user's emotional state, which can increase the burden on staff and reduce customer satisfaction. This invention provides a system that recognizes the user's emotional state and appropriately suggests and executes tasks based on that state, thereby improving the efficiency of brick-and-mortar stores and the customer experience.

[1030] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1031] In this invention, the server includes means for extracting text information from an image acquired by a camera using a character recognition engine, means for analyzing the extracted text information and surrounding background information to identify the next suggested task to be performed, means for detecting the user's facial expression and tone of voice to recognize emotions, and means for sorting the suggested tasks in order of priority based on the emotional information, thereby making it possible to suggest and execute appropriate tasks based on the emotional states of store staff and customers.

[1032] A "character recognition engine" is a device or program that analyzes and extracts character information from images acquired by a camera.

[1033] "Surrounding background information" is a general term for accompanying data and environmental information related to text information.

[1034] A "proposed task" is a proposal for the next work to be done based on the analyzed and parsed information.

[1035] "User notification means" refers to the means by which the proposed task is communicated to the user, displayed as a notification or alert.

[1036] A "task executor" is a processing mechanism for executing a suggested task based on user input.

[1037] "Facial expression detection" is a process that uses a camera to analyze a user's facial expressions and recognize their emotions.

[1038] "Tone of voice detection" is a process that uses a microphone to analyze the tone of a user's voice and recognize their emotions.

[1039] "Emotion recognition" is the process of identifying a user's emotions from facial expressions and tone of voice.

[1040] The "means for sorting by priority" is a processing function that rearranges the order of proposed tasks based on their importance and ease of execution.

[1041] "Multi-modality artificial intelligence" is an artificial intelligence system that combines and analyzes multiple data formats (for example, text information and background information) to provide users with appropriate information.

[1042] This invention uses a "smart store assistant" system in a physical store to efficiently and effectively suggest tasks to store staff and customers and assist them in carrying them out. A specific embodiment of this system will be described below.

[1043] First, the device uses a camera to scan the barcode or label of a product in a physical store. This device can be a smartphone or smart glasses. The camera is activated and the captured image is temporarily stored on the device. The image is then sent to a server for pre-processing. This pre-processing involves removing noise from the image and adjusting its resolution to improve the accuracy of character recognition by the OCR engine.

[1044] Next, the server inputs the preprocessed image data into an OCR engine, such as Tesseract OCR or Google Cloud Vision API. This extracts the text information from the image and records it as text data. At this stage, information such as the product name, price, and expiration date are obtained.

[1045] The extracted text information is input into a multi-modality AI system. The server analyzes the text information in combination with the surrounding contextual information to identify tasks to be proposed. For example, this could include suggesting restocking of low-stock items or proposing promotions for specific products.

[1046] The device detects the user's facial expressions and tone of voice through a camera and microphone, and analyzes them with an emotion recognition engine. The recognized emotion information is sent to the server and reflected in task suggestions. For example, if the user is tired, the device will prioritize simple tasks to reduce the user's burden.

[1047] Finally, the server sends the prioritized task list to the device. The device notifies the user of the task list through a pop-up window or an alert. When the user selects one of the suggested tasks, the task is automatically executed. For example, it may send a stock replenishment request or start a sales promotion campaign. Once the task is completed, the device feeds back the results to the server. The server stores this in a database and uses it to improve the accuracy of future suggestions.

[1048] For example, here's a prompt to notify you when an item is low in stock:

[1049] Scan the receipt image.

[1050] Important Task: This item is low on stock and needs to be restocked. Would you like to submit a restock request?

[1051] This system will reduce the burden on store staff in physical stores, enable them to carry out their work more efficiently, and improve customer satisfaction.

[1052] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1053] Step 1:

[1054] The device uses a camera to scan products or labels in a physical store. The user picks up the device (smartphone or smart glasses) and points the camera at the target product to capture an image. In this step, the camera is activated and the captured image is temporarily saved in the device. The image is then sent to the server. The input is the image of the product or label, and the output is the image data sent to the server.

[1055] Step 2:

[1056] The server preprocesses the received image data. Preprocessing includes noise removal and resolution adjustment. Specifically, the server converts the image to grayscale and applies Gaussian blur to remove noise. It also adjusts the resolution appropriately to improve the accuracy of character recognition. The input is the received raw image data, and the output is preprocessed, high-quality image data.

[1057] Step 3:

[1058] The server inputs the preprocessed image data into the OCR engine. The OCR engine (for example, Tesseract OCR or Google Cloud Vision API) analyzes the character information in the image and extracts it as text data. In this step, the server calls the OCR engine to perform the analysis and saves the obtained text data on the server. The input is the preprocessed image data, and the output is text data containing character information.

[1059] Step 4:

[1060] The server inputs the extracted text information and surrounding background information into a multi-modality AI for analysis. This identifies suggested tasks that are highly relevant to the user. The server then integrates the text information and background information and analyzes it using an AI model. As a result, a list of suggested tasks is generated. The input is text information and background information, and the output is a list of suggested tasks.

[1061] Step 5:

[1062] The device detects the user's facial expressions and tone of voice through a camera and microphone, and analyzes them using an emotion recognition engine. When the user operates the device, the camera captures the user's facial expressions and the microphone records the user's tone of voice. This data is sent from the device to a server and analyzed by the emotion recognition engine. The input is facial expression and voice data, and the output is recognized emotional information.

[1063] Step 6:

[1064] The server re-evaluates the proposed task list based on the emotion information and sorts it in order of priority. The server receives the emotion recognition results and adjusts the task priority according to the user's state. For example, if the user is tired, it prioritizes easier tasks. The input is the proposed task list and emotion information, and the output is a task list sorted in order of priority.

[1065] Step 7:

[1066] The terminal notifies the user of a priority-ordered task list sent from the server. The terminal displays the task list in a pop-up window or alert format, allowing the user to select from the list. When the user selects a specific task, that task is executed. The input is a task list sorted by priority, and the output is a notification to the user and the execution of the task.

[1067] Step 8:

[1068] After completing a task, the device sends the result (success, failure, feedback) to the server. The server stores the received feedback in a database and uses it as training data for the AI ​​model to improve the accuracy of the next task proposal. The input is the task execution result and feedback, and the output is the feedback information stored as learning data.

[1069] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1070] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1071] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1072] [Fourth embodiment]

[1073] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1074] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1075] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1076] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1077] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1078] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1079] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1080] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1081] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1082] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1083] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1084] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1085] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1086] An embodiment of the present invention will be described below. The present invention provides an OCR application that uses a camera to recognize characters, analyzes surrounding information, and suggests and executes the next task to be performed.

[1087] 1. Camera image acquisition and preprocessing

[1088] The device activates the camera and captures the image taken by the user. For example, the user activates a scanner application and takes a picture of a receipt or document with the camera. The image is temporarily stored on the device and then sent to the server. The server preprocesses the received image, removing noise and optimizing the resolution to improve the accuracy of character recognition.

[1089] Examples:

[1090] The user takes a photo of the receipt, and the image is sent from the device to a server where noise is removed and resolution is adjusted.

[1091] 2. Character Recognition Processing

[1092] The server inputs the preprocessed image data into a character recognition engine. The OCR engine (e.g., an open-source OCR tool or a cloud-based OCR API) recognizes the characters in the image and extracts them as text data. The extracted text is then checked for spelling and grammar before being formatted into accurate data.

[1093] Examples:

[1094] The server extracts text information such as the product name, price, and store name from the receipt image.

[1095] 3. Analysis of surrounding information and task proposal

[1096] The server inputs the text information extracted by the OCR engine and the surrounding background information into a multi-modality AI. The AI ​​analyzes this information and identifies the most relevant tasks for the user. As a result of the analysis, a task list is generated and sorted by priority. This task list is then converted into a format that can be displayed to the user and sent to the device.

[1097] Examples:

[1098] The server uses the extracted receipt information and background information to suggest tasks such as "register this expenditure in a household accounting app" or "save a coupon."

[1099] 4. User Notification and Task Execution

[1100] The device notifies the user of the task list sent from the server. The notification is displayed in the form of a pop-up window or an alert. When the user selects one of the suggested tasks, the device executes that task. For example, it may register data in a household accounting app or launch an email sending app.

[1101] Examples:

[1102] When the user selects "Register in household accounting app," the device opens the household accounting app and enters the receipt information.

[1103] 5. Feedback of results

[1104] Once the task is completed, the device sends the result (success, failure, feedback) to the server, which stores the received feedback in a database and uses it as training data for the AI ​​model to improve the accuracy of the next task suggestions.

[1105] Examples:

[1106] If the data is successfully registered in the household accounting app, the information is sent to the server and used for analyzing receipt information the next time.

[1107] This embodiment allows users to efficiently handle various tasks with one OCR application, eliminating the need to use multiple dedicated applications. Furthermore, the use of multi-modality artificial intelligence improves the accuracy of suggested tasks and makes user operation easier and more intuitive.

[1108] The processing flow will be explained below.

[1109] Step 1:

[1110] The user launches the OCR application and switches to camera mode. The user presses the capture button to capture an image.

[1111] Step 2:

[1112] The captured image is temporarily stored on the device and then sent to the server.

[1113] Step 3:

[1114] The server decodes the received image into a file format and performs preprocessing such as adjusting the resolution and removing noise.

[1115] Step 4:

[1116] The server inputs the preprocessed image data into a character recognition engine, and the OCR engine extracts character information from the image.

[1117] Step 5:

[1118] The server performs spelling and grammar checks on the extracted text information and formats it into accurate text data.

[1119] Step 6:

[1120] The server inputs the text information and surrounding background information into the multi-modality artificial intelligence, and the AI ​​begins its analysis.

[1121] Step 7:

[1122] Based on the analysis results, the server's AI generates suggested tasks that are most relevant to the user and creates a task list sorted by priority.

[1123] Step 8:

[1124] The server formats the generated task list and sends it to the terminal for notification to the user.

[1125] Step 9:

[1126] The device displays the task list to the user as a popup window or alert, and the user selects one of the suggested tasks.

[1127] Step 10:

[1128] The device executes the corresponding task based on the user's selection, for example, launching a household accounting app and entering text data.

[1129] Step 11:

[1130] The server receives the task execution results (success, failure, feedback) from the terminal and stores them in a database.

[1131] Step 12:

[1132] The server uses the saved result data as learning data for the AI ​​and updates the model to improve the accuracy of the next proposed task.

[1133] Example 1

[1134] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1135] Conventional OCR applications have insufficient character recognition accuracy, often requiring users to manually correct information. Furthermore, information analysis and task suggestions after character recognition are ineffective, resulting in poor user convenience. Furthermore, the accuracy of the suggested tasks is low, and there is a lack of a learning mechanism that utilizes feedback.

[1136] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1137] In this invention, the server includes means for temporarily saving images acquired from the camera in an internal storage device, means for transmitting images from the internal storage device to the server and performing noise reduction and resolution optimization, means for using a character recognition engine to extract character information from the noise-removed and resolution-optimized images, means for analyzing the extracted character information and surrounding background information using multi-modality artificial intelligence and identifying a suggested next task to be performed, means for notifying the user of the suggested task, means for executing the suggested task based on user input, and means for sending feedback after task execution to the server and storing the feedback in a database to use as training data. This enables improved character recognition accuracy, effective information analysis and task suggestions, and a learning mechanism using feedback.

[1138] A "camera" is a device for taking still images and videos.

[1139] "Internal storage" means a storage medium within a device that is used to store data temporarily or permanently.

[1140] A "server" is a computer system that provides services to other computers on a network.

[1141] "Noise reduction" is the process of removing unwanted interference and noise from image and audio data.

[1142] "Resolution optimization" is a process of adjusting the image so that the details can be displayed clearly.

[1143] A "character recognition engine" is software or an algorithm that identifies characters in an image and extracts them as text data.

[1144] "Multi-modality artificial intelligence" is artificial intelligence that integrates, understands, and analyzes data in multiple different formats (e.g., text and images).

[1145] "Analysis" is the process of examining data in detail to understand its meaning and structure.

[1146] A "suggested task" is a specific action or task that is shown to the user to take next.

[1147] "Notification means" refers to a method or device for notifying a user of information.

[1148] "Feedback" refers to reporting the results of a task and user evaluation information to the server, and is used for future improvements.

[1149] A "database" is a system for efficiently storing, managing, and searching data.

[1150] An embodiment of the present invention will be described in detail. The invention provides an OCR application that uses a camera to recognize characters, analyzes surrounding information, and proposes and executes the next task to be performed. The following describes the flow of data processing and data calculation using each hardware and software.

[1151] 1. Camera image acquisition and preprocessing

[1152] The device activates the camera and acquires the image taken by the user. For example, the user activates a scanner application and takes a picture of a receipt or document with the camera. This image is temporarily stored in the device's internal storage and then sent to the server. The server preprocesses the received image, removing noise and optimizing the resolution to improve the accuracy of character recognition. For example, it uses the image processing library "OpenCV" to remove noise and optimizes the resolution using the "PhotoShop API."

[1153] Examples:

[1154] The user takes a photo of the receipt, and the image is sent from the device to a server where noise is removed and resolution is adjusted.

[1155] 2. Character Recognition Processing

[1156] The server inputs the preprocessed image data into a character recognition engine. The OCR engine used can be the open source "Tesseract" or the cloud-based "Google Cloud Vision API." The server calls the OCR engine, recognizes the characters in the image, and extracts them as text data. The extracted text is then checked for spelling and grammar using a natural language processing tool (such as "NLTK" or "SpaCy"), and formatted into accurate text data.

[1157] Examples:

[1158] The server extracts text information such as the product name, price, and store name from the receipt image.

[1159] 3. Analysis of surrounding information and task proposal

[1160] The server inputs the text information extracted by the OCR engine and the surrounding background information into a multi-modality AI (such as GPT-3 or BERT). The AI ​​analyzes this information and identifies the most relevant tasks for the user. As a result of the analysis, a task list is generated and sorted by priority. This task list is then converted into a format that can be displayed to the user and sent to the device.

[1161] Examples:

[1162] The server uses the extracted receipt information and background information to suggest tasks such as "register this expenditure in a household accounting app" or "save a coupon."

[1163] 4. User Notification and Task Execution

[1164] The device notifies the user of the task list sent from the server. The notification is displayed in the form of a pop-up window or an alert. When the user selects one of the suggested tasks, the device executes that task. Specifically, it registers data in a household accounting app or launches an email sending app.

[1165] Examples:

[1166] When the user selects "Register in household accounting app," the device opens the household accounting app and inputs the receipt information.

[1167] 5. Feedback of results

[1168] Once the task is completed, the device sends the result (success, failure, feedback) to the server, which stores the received feedback in a database and uses it as training data for the AI ​​model to improve the accuracy of the next task suggestions.

[1169] Examples:

[1170] If the data is successfully registered in the household accounting app, the information is sent to the server and used for analyzing receipt information the next time.

[1171] Prompt Sentence Examples

[1172] In a system for implementing the invention, more specific task suggestions can be made by using prompt sentences such as the following to input to a generative AI model:

[1173] example:

[1174] "I took a photo of my receipt. Can you give me some suggestions to help me manage my household finances and coupons?"

[1175] The above is an embodiment of the present invention. This embodiment allows users to efficiently handle a variety of tasks with a single OCR application, eliminating the need to use multiple dedicated applications. Furthermore, by utilizing multi-modality artificial intelligence, the accuracy of the suggested tasks is improved, and user operation becomes simpler and more intuitive.

[1176] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1177] Step 1:

[1178] A user starts a scanner application on their device and uses the camera to take a picture of a receipt or document. At this time, the device's camera captures the image data and temporarily stores it in the device's internal storage. The input is the receipt image taken by the user, and the output is the image data stored in the internal storage.

[1179] Step 2:

[1180] The device sends image data stored in its internal storage device to the server. The image data is sent over the Internet using a secure communication protocol such as HTTPS. The input is the image data in the internal storage device, and the output is the image data sent to the server.

[1181] Step 3:

[1182] The server analyzes the image data it receives and performs preprocessing to remove noise and optimize resolution. Specifically, it removes noise using the image processing library "OpenCV" and optimizes resolution using "PhotoShop API." The input is the image data sent to the server, and the output is image data with noise removed and resolution optimized.

[1183] Step 4:

[1184] The server inputs the preprocessed image data into a character recognition engine. The OCR engine used is "Tesseract" or "Google Cloud Vision API." The server uses the OCR engine to recognize characters in the image and extract them as text data. The input is noise-removed and resolution-optimized image data, and the output is the extracted text data.

[1185] Step 5:

[1186] The server uses natural language processing tools (such as NLTK and SpaCy) to check the spelling and grammar of the extracted text data, which then converts it into accurate text data. The input is text data obtained from the OCR engine, and the output is accurate text data that has been checked for spelling and grammar.

[1187] Step 6:

[1188] The server inputs the text data extracted by the OCR engine and the surrounding context into a multi-modality AI model (e.g., GPT-3 or BERT). The AI ​​model analyzes this information and identifies tasks that are highly relevant to the user. The input is the formatted text data and context, and the output is a list of suggested tasks.

[1189] Step 7:

[1190] The server converts the generated task list into a notification format and sends it to the terminal. For example, it constructs the task list in JSON format and sends it to the terminal via the Internet. The input is the specified task list, and the output is the task list sent to the terminal.

[1191] Step 8:

[1192] The terminal analyzes the task list received from the server and notifies the user. The notification is displayed in the form of a pop-up window or an alert. The input is the task list sent from the server, and the output is the tasks to be notified to the user.

[1193] Step 9:

[1194] The user selects one of the proposed tasks, for example, registering data in a household accounting app. The input is the notified task list, and the output is the specific task selected.

[1195] Step 10:

[1196] The device executes the selected task. Specifically, it opens a household accounting app and automatically inputs the receipt information the user wants to register. The input is the selected task information, and the output is the executed task.

[1197] Step 11:

[1198] The terminal sends the result (success, failure, feedback) after completing the task to the server. The result data is sent to the server via the Internet. The input is the result of the task execution, and the output is the feedback data sent to the server.

[1199] Step 12:

[1200] The server stores the received feedback in a database. The feedback data is used as training data for the AI ​​model to improve the accuracy of the next task suggestion. The input is the feedback data sent from the device, and the output is the updated database information.

[1201] (Application example 1)

[1202] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1203] In physical stores, there is a need for support to enable users to quickly and accurately obtain product information and choose appropriate actions based on that information. However, current systems require manual information collection and analysis, which is extremely time-consuming. In addition, there is a lack of systems that allow users to easily compare prices and product reviews at other stores while shopping in a store. Therefore, technology is needed to enable users to shop efficiently and effectively.

[1204] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1205] In this invention, the server includes means for extracting text information from an image acquired by a camera using a character recognition engine, means for analyzing the extracted text information and surrounding background information and identifying a suggested next task to be performed, means for notifying the user of the suggested task, means for executing the suggested task based on the user's input, means for performing preprocessing such as image noise reduction and resolution optimization, means for performing cloud-based analysis, and means for suggesting tasks such as comparing product prices and displaying product reviews. This enables users to make optimal shopping decisions in real time based on product and shelf information captured by the camera.

[1206] A "character recognition engine" is software or hardware that recognizes characters from images acquired from a camera or other imaging device and extracts them as digital text data.

[1207] "Preprocessing" refers to processing of the captured image, such as noise removal and resolution optimization, to improve the accuracy of character recognition.

[1208] "Cloud-based analysis" is a technology that uses computing resources in a cloud environment to analyze and calculate data, enabling high-speed, large-scale data processing.

[1209] "Multi-modality AI" is an AI technology that integrates multiple different data modalities (e.g., text, images, audio), analyzes them comprehensively, and makes highly accurate task suggestions and predictions.

[1210] A "suggested task" is the next action or task that the AI ​​identifies based on user input and acquired data.

[1211] "Noise reduction" is a process for removing unnecessary information and data in image preprocessing to improve image quality.

[1212] "Resolution optimization" is a process that optimizes the fine details of an image to improve the accuracy of character recognition.

[1213] "User notification" means the notification method used by the system to inform the user of required information or tasks, including pop-up windows and alerts.

[1214] "Task execution" means actually performing the proposed task selected by the user, which includes launching an application and registering data.

[1215] "Price comparison" refers to the process of collecting and comparing price information for a specific product at different sales locations and times to provide users with the best purchasing options.

[1216] "Review display" refers to displaying ratings and impressions posted by other users about a specific product, allowing users to refer to other people's opinions before purchasing.

[1217] The system for implementing the present invention includes the following detailed processes to support users' shopping activities.

[1218] The programs implemented on the server and terminal first include a means for acquiring and preprocessing camera images. The terminal's camera is activated, and the user takes a photo of the product or shelf information. The captured image is temporarily saved on the terminal and then sent to the server. The server uses the image processing library OpenCV (cv2) to remove noise and optimize the resolution of the received image data. After preprocessing, the image is input into the character recognition engine Tesseract (pytesseract), and character information such as the product name, price, and features is extracted as text data.

[1219] The extracted text data and surrounding contextual information are analyzed by a multi-modality AI on the server to identify new tasks. This AI model performs cloud-based analysis using the Google Cloud Vision API to generate optimal suggested tasks for the user (e.g., comparing product prices or displaying reviews) and sort them by priority. The resulting task list is notified to the user and sent to the device. Notifications are displayed as pop-up windows or alerts on the device.

[1220] The user checks the notification and selects one of the suggested tasks. For example, if they select the task of comparing prices of a specific product with other stores, the device will launch a web browser and display the relevant search page. For other tasks, such as displaying product reviews, the device will display review information from appropriate sites. This process allows the user to efficiently proceed with their shopping.

[1221] Furthermore, the results of the task execution are sent to the server as feedback and saved as learning data to improve the accuracy of the next task proposal. The server uses this feedback to continuously improve the accuracy of the generative AI model.

[1222] Specific prompt examples:

[1223] Prompt after taking a photo of the receipt:

[1224] Your receipt information has been extracted. What do you want to do with this information? Choose from the following options:

[1225] 1. Register for a household accounting app

[1226] 2. Save the coupon

[1227] 3. Compare prices with other stores

[1228] Product comparison prompt:

[1229] Enter the name of the product you want to search for:

[1230] The system allows users to efficiently gather product information and compare prices based on captured images, enabling them to make the best purchasing decision. Specific hardware components include smartphones and smart glasses, while software components include OpenCV, Tesseract, Google Cloud Vision API, and multi-modality artificial intelligence.

[1231] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1232] Step 1: Acquire camera images

[1233] The device activates the camera, and the user takes a picture of the product or shelf information. The input is image data acquired from the device's camera, and the output is an image file temporarily stored in the device. This image file is used in the next processing step.

[1234] Step 2: Image preprocessing

[1235] The device sends the saved image file to the server. The server performs noise removal and resolution optimization on the received image data. The input is the image data sent from the device, and the output is a preprocessed image that has been noise removed and resolution optimized. This process uses an image processing library called OpenCV.

[1236] Step 3: Character Recognition

[1237] The server inputs the preprocessed image data into the OCR engine (Tesseract). The OCR engine recognizes the characters in the image and extracts them as text data. The input is the preprocessed image data, and the output is the extracted text data. Specifically, it detects the outlines of the characters in the image and recognizes them as characters.

[1238] Step 4: Analyze text and context

[1239] The server inputs the text information extracted by the OCR engine and the surrounding background information into a multi-modality artificial intelligence. The AI ​​analyzes this information and suggests related tasks to the user (for example, comparing product prices or displaying reviews). The input is the extracted text data and background information, and the output is a task list as the analysis result. Specifically, it extracts product features from the text data and generates related tasks based on this.

[1240] Step 5: Notify users

[1241] The server sends the generated task list to the terminal and notifies the user. The notification is displayed on the terminal as a pop-up window or alert. The input is the task list as the analysis result, and the output is the notification to the user. Specifically, it lists tasks in order of priority based on specific conditions and presents this to the user.

[1242] Step 6: User Task Selection

[1243] The user selects one of the suggested tasks. The input is the task list presented as a notification, and the output is the user's selection. The specific behavior involves selecting a task through a user interface.

[1244] Step 7: Execute the task

[1245] The device executes the selected task based on the user's input. For example, if a user wants to compare the prices of a particular product with other stores, the device will launch a web browser and display the relevant page. If a user wants to view product reviews, the device will display the review information from the appropriate website. The input is the user's selection, and the output is the result of each task. Specific operations include launching an application or displaying data according to the selected task.

[1246] Step 8: Feedback on results

[1247] The results of the task execution are sent from the device to the server. The server stores this feedback in a database and uses it as training data to improve the accuracy of the next task suggestion. The input is the task execution result, and the output is a database entry for the next training. Specific operations include evaluating the execution results and recording them.

[1248] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1249] An embodiment of the present invention is described below. The present invention provides a system that combines an OCR application that uses a camera to recognize characters, analyzes peripheral information, and suggests and executes the next task to be performed with an emotion engine that recognizes the user's emotions.

[1250] 1. Camera image acquisition and preprocessing

[1251] The device activates the camera and captures the image taken by the user. For example, the user launches an application and takes a picture of a receipt or document with the camera. The image is temporarily stored on the device and then sent to the server. The server preprocesses the received image, removing noise and optimizing the resolution to improve the accuracy of character recognition.

[1252] Examples:

[1253] The user takes a photo of the receipt, and the image is sent from the device to a server where noise is removed and resolution is adjusted.

[1254] 2. Character Recognition Processing

[1255] The server inputs the preprocessed image data into a character recognition engine. The OCR engine (e.g., an open-source OCR tool or a cloud-based OCR API) recognizes the characters in the image and extracts them as text data. The extracted text is then checked for spelling and grammar before being formatted into accurate data.

[1256] Examples:

[1257] The server extracts text information such as the product name, price, and store name from the receipt image.

[1258] 3. Analysis of surrounding information and task proposal

[1259] The server inputs the text information extracted by the OCR engine and the surrounding background information into a multi-modality AI. The AI ​​analyzes this information and identifies the most relevant tasks for the user. As a result of the analysis, a task list is generated and sorted by priority. This task list is then converted into a format that can be displayed to the user and sent to the device.

[1260] Examples:

[1261] The server uses the extracted receipt information and background information to suggest tasks such as "register this expenditure in a household accounting app" or "save a coupon."

[1262] 4. User Emotion Recognition

[1263] The device detects the user's facial expressions and tone of voice through the camera and microphone, and recognizes the user's emotions using an emotion engine. For example, when a user takes a photo of a receipt, the camera detects the user's smile. This information is sent from the device to the server.

[1264] Examples:

[1265] When a user takes a photo of a receipt using their smartphone camera, the emotion engine recognizes the user's smile (happiness) emotion.

[1266] 5. Emotion-Based Task Adjustment

[1267] The server re-evaluates the task list based on the emotional information recognized by the emotion engine and prioritizes tasks that suit the user's emotions. For example, if the user looks tired, it will prioritize easy tasks.

[1268] Examples:

[1269] The emotion engine detects when the user has a tired expression, and the server prioritizes and suggests a simplified task such as "immediately record this expenditure in a household accounting app."

[1270] 6. User Notification and Task Execution

[1271] The device notifies the user of the task list sent from the server. The notification is displayed in the form of a pop-up window or an alert. When the user selects one of the suggested tasks, the device executes that task. For example, it may register data in a household accounting app or launch an email sending app.

[1272] Examples:

[1273] When the user selects "Register in household accounting app," the device opens the household accounting app and enters the receipt information.

[1274] 7. Feedback of results

[1275] Once the task is completed, the device sends the result (success, failure, feedback) to the server. The server stores the received feedback in a database and uses it as training data for the AI ​​model to improve the accuracy of the next task suggestions. Emotion recognition results are also stored as learning data and used to improve the user experience.

[1276] Examples:

[1277] If the data is successfully registered in the household accounting app, the information is sent to the server and used for analyzing receipt information the next time. Also, if the user smiles with satisfaction after completing the task, the emotional information is also saved.

[1278] This embodiment allows users to efficiently handle a variety of tasks with a single OCR application, eliminating the need to use multiple dedicated applications. Furthermore, the combination of emotion engines enables optimal task suggestions based on the user's emotional state, resulting in a more intuitive and satisfying user experience.

[1279] The processing flow will be explained below.

[1280] Step 1:

[1281] The user launches the OCR application and switches to camera mode. The user presses the capture button to capture an image.

[1282] Step 2:

[1283] The captured image is temporarily stored on the device and then sent to the server.

[1284] Step 3:

[1285] The server decodes the received image and performs pre-processing such as adjusting the resolution and removing noise.

[1286] Step 4:

[1287] The server inputs the preprocessed image data into a character recognition engine, and the OCR engine extracts character information from the image.

[1288] Step 5:

[1289] The server performs spelling and grammar checks on the extracted text information and formats it into accurate text data.

[1290] Step 6:

[1291] The server inputs the text information and surrounding background information into the multi-modality artificial intelligence, and the AI ​​begins its analysis.

[1292] Step 7:

[1293] Based on the analysis results, the server's AI generates suggested tasks that are most relevant to the user and creates a task list sorted by priority.

[1294] Step 8:

[1295] The server formats the generated task list and sends it to the terminal for notification to the user.

[1296] Step 9:

[1297] The device displays the task list to the user as a popup window or alert, and the user selects one of the suggested tasks.

[1298] Step 10:

[1299] The device executes the corresponding task based on the user's selection, for example, launching a household accounting app and entering text data.

[1300] Step 11:

[1301] The device detects the user's facial expressions and tone of voice through the camera and microphone, and recognizes the user's emotions using an emotion engine.

[1302] Step 12:

[1303] The device sends the user's emotional information to the server, which then reevaluates and adjusts the task list based on the emotional information.

[1304] Step 13:

[1305] The server transmits the task list readjusted based on the emotion information to the terminal, and the terminal notifies the user again.

[1306] Step 14:

[1307] If the user selects the suggested task again, the device will execute the adjusted task, for example, prioritizing the execution of the easier task.

[1308] Step 15:

[1309] After the task is completed, the device sends the result (success, failure, feedback) to the server.

[1310] Step 16:

[1311] The server stores the results it receives in a database and uses them as training data for the AI ​​model to improve the accuracy of the next task proposal, including the emotion recognition results.

[1312] Example 2

[1313] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1314] In modern society, many users need to quickly and accurately manage various documents and payment records in their daily lives. Conventional applications using optical character recognition (OCR) technology only provide character recognition and simple data conversion, but lack the ability to manage tasks or suggest tasks based on the user's emotional state. In particular, task suggestions that take into account the user's mental and emotional state are crucial for improving the user experience. Therefore, there is a need for a system that allows users to efficiently manage documents and receive appropriate task suggestions based on their emotional state.

[1315] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1316] In this invention, the server includes means for activating a camera to acquire an image, means for preprocessing the acquired image to remove noise and optimize resolution, means for using a character recognition engine to extract character information from the preprocessed image, means for analyzing the extracted character information and surrounding background information to identify a next suggested task to be performed, means for using an emotion engine to recognize the user's emotions, means for reevaluating the task list based on the recognized emotion information, means for notifying the user of the suggested tasks, means for executing the suggested tasks based on the user's input, and means for recording the execution results and emotion information and evaluating them to improve the accuracy of the next suggestion. This enables the user to efficiently and accurately perform a series of document management and task suggestions, and further enables the user to receive optimal task suggestions adapted to their emotional state.

[1317] "Means for activating the camera" refers to a device or software that has the functionality to allow a user to activate the camera and take an image by operating an application.

[1318] "Means for preprocessing images" refers to devices or software that have the function of removing noise and optimizing resolution from acquired images, and performing processing to improve the accuracy of character recognition.

[1319] A "character recognition engine" is software or algorithms that use OCR technology to extract character information from pre-processed image data.

[1320] "Surrounding background information" refers to information other than textual information contained in an image, such as location information or user history information, and is additional data used to suggest tasks.

[1321] An "emotion engine" is software or algorithms that use a camera or microphone to analyze facial expressions and tone of voice in order to recognize a user's emotional state (e.g., joy or fatigue).

[1322] A "suggested task" is a suggestion to the user about the next action or process to be taken based on the text information and surrounding information analyzed by the system.

[1323] The "means for reevaluating the task list" refers to a function that readjusts the priority of an already generated task list based on the user's emotional information and performs the process of suggesting the most suitable task list to the user.

[1324] A "notification mechanism" is any device or software capable of displaying information in the form of a pop-up window, alert, or other form to inform the user of a proposed task.

[1325] A "means for performing a task" is any device or software that has the functionality to actually perform the proposed task selected by the user.

[1326] "Means for recording execution results" refers to devices or software that have the ability to record the results (success, failure, feedback) after a task is completed and store them in a database to improve the accuracy of suggestions next time.

[1327] An embodiment of the present invention is described below. The present invention provides a system that combines an OCR application that uses a camera to recognize characters, analyzes peripheral information, and suggests and executes the next task to be performed with an emotion engine that recognizes the user's emotions.

[1328] Camera image acquisition and preprocessing

[1329] The device activates the camera and acquires an image taken by the user. For example, the user launches an application and takes a picture of a receipt or document with the camera. This image is temporarily stored on the device and then sent to the server. The server then preprocesses the received image, removing noise and optimizing the resolution to improve the accuracy of character recognition. Specifically, it uses noise reduction algorithms such as Gaussian Blur and image resolution optimization techniques.

[1330] Example: A user takes a photo of a receipt, the image is sent from the device to a server, which removes noise from the image and scales it to the appropriate resolution.

[1331] Character Recognition Processing

[1332] The server inputs the preprocessed image data into a character recognition engine. An OCR engine (such as the open-source Tesseract or the cloud-based Google Cloud Vision API) recognizes the characters in the image and extracts them as text data. The extracted text is then checked for spelling and grammar to ensure accurate data is generated.

[1333] Example: The server extracts text information such as "product name," "price," and "store name" from a receipt image and performs spell checking and grammar correction.

[1334] Peripheral information analysis and task suggestions

[1335] The server inputs the text information extracted by the OCR engine and the surrounding background information into a multi-modality AI. The AI ​​analyzes this information and identifies the most relevant tasks for the user. As a result of the analysis, a task list is generated and sorted by priority. The generated task list is converted into a notification format and sent to the device.

[1336] Example: The server uses the extracted receipt information and background information to identify tasks such as "record this expenditure in a household accounting app" or "save a coupon" and create a list.

[1337] User emotion recognition

[1338] The device detects the user's facial expressions and tone of voice through the camera and microphone, and recognizes the user's emotions using an emotion engine. For example, it uses machine learning models for emotion recognition (Microsoft Face API or Emotion API). The recognized emotion information is sent from the device to a server.

[1339] Example: When a user takes a picture of a receipt using the smartphone camera, the emotion recognition engine recognizes the user's smile (happiness) emotion and sends that information to the server.

[1340] Emotion-based task adjustment

[1341] The server reevaluates the task list based on the emotional information received. It then adjusts the priority of tasks based on the recognized emotional information and suggests the best possible tasks for the user. For example, if the user looks tired, the list is adjusted to prioritize easier tasks.

[1342] Example: If the emotion engine detects that the user has a tired expression, the server will prioritize and suggest simplified tasks.

[1343] User notification and task execution

[1344] The device notifies the user of the task list sent from the server. The notification is displayed in the form of a pop-up window or an alert, and when the user selects one of the suggested tasks, the device executes that task. For example, it may register data in a household accounting app or launch an email sending app.

[1345] Example: When a user selects "Register in household accounting app," the device opens the household accounting app and enters receipt information.

[1346] Feedback of results

[1347] Once the task is completed, the device sends the result (success, failure, feedback) to the server. The server stores the received feedback in a database and uses it as training data for the AI ​​model to improve the accuracy of the next task suggestion. The server also stores the emotion recognition results as learning data to help improve the user experience.

[1348] Example: When data is successfully registered in a household accounting app, the information is sent to the server and used for analyzing receipt information the next time. If the user smiles with satisfaction after completing a task, the emotional information is also stored on the server.

[1349] Example prompts to input to the generative AI model

[1350] "Please explain in detail the system that analyzes receipt information and suggests appropriate tasks for the user."

[1351] "Please explain the specific steps in the process flow of a system that recognizes a user's emotions and adjusts tasks based on those emotions."

[1352] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1353] Step 1:

[1354] The device starts the camera and acquires the image taken by the user. The user starts the application and takes a picture of a receipt or document with the camera. As a result, the image data is temporarily stored in the device's local memory. The input data is the camera image, and the output data is an image file stored on the device.

[1355] Step 2:

[1356] The device sends the captured image to the server. Here, the device transfers the image data to the server using a secure communication protocol (e.g., HTTPS). The input data is the image file stored in the device, and the output data is the image file sent to the server. Specifically, data is transferred using an HTTP request.

[1357] Step 3:

[1358] The server preprocesses the images received. Preprocessing includes noise removal (for example, applying Gaussian Blur) and resolution optimization. The input data is the image file received by the server, and the output data is the preprocessed image data. Specifically, the image processing algorithm is applied using Python's OpenCV library.

[1359] Step 4:

[1360] The server inputs the preprocessed image data into a character recognition engine. An OCR engine (e.g., Tesseract) is used to recognize the characters in the image and extract them as text data. The input data is the preprocessed image data, and the output data is character string data (text). Specifically, the OCR engine is called to perform image analysis and the recognition results are obtained as text.

[1361] Step 5:

[1362] The server performs spelling and grammar checks on the extracted text data. It uses natural language processing tools (e.g., NLTK or SpaCy) to analyze the text data and format it into accurate data. The input data is text data obtained from the OCR engine, and the output data is accurately formatted text data. Specifically, it performs grammar and spelling checks using regular expressions and machine learning models.

[1363] Step 6:

[1364] The server inputs the text information extracted by the OCR engine and the surrounding background information into a multi-modality AI. The multi-modality AI analyzes this information and identifies tasks that are highly relevant to the user. The input data is precisely formatted text data and surrounding information, and the output data is a task list sorted by priority. Specifically, the data is input into the AI ​​model and task relevance is calculated.

[1365] Step 7:

[1366] The server converts the generated task list into a notification format and sends it to the terminal. The input data is the task list, and the output data is in a notification data format (e.g., JSON format). Specific operations include data formatting and sending operations.

[1367] Step 8:

[1368] The terminal notifies the user of the task list sent from the server. The notification is displayed as a pop-up window or an alert. The input data is the notification data received from the server, and the output data is the notification visually displayed to the user. Specifically, the notification is displayed using GUI (Graphical User Interface) elements.

[1369] Step 9:

[1370] The user selects one of the proposed tasks. The input data is the presented task list, and the output data is the task selected by the user. The selection is made by the user's action.

[1371] Step 10:

[1372] The device executes a task selected by the user. For example, it registers data in a household accounting app or launches an email sending app. The input data is the task selected by the user, and the output data is the task completion status (success or failure). The specific operation is to call the API of the corresponding application.

[1373] Step 11:

[1374] When the task is completed, the device sends the result (success, failure, feedback) to the server. The input data is the task completion status, and the output data is the feedback data. Specifically, the feedback is sent to the server using an HTTP request.

[1375] Step 12:

[1376] The server stores the received feedback in a database and uses it to improve the accuracy of the next task suggestion. It also stores the emotion recognition results and uses them to improve the user experience. The input data is the feedback data and the emotion recognition data, and the output data is the updated database entry. Specifically, the system stores and updates data using database queries.

[1377] (Application example 2)

[1378] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1379] In modern brick-and-mortar stores, store staff must manage multiple tasks simultaneously, requiring efficient operations. They are also required to provide high-quality service to customers, making store management more complex. Conventional systems suggest tasks without considering the user's emotional state, which can increase the burden on staff and reduce customer satisfaction. This invention provides a system that recognizes the user's emotional state and appropriately suggests and executes tasks based on that state, thereby improving the efficiency of brick-and-mortar stores and the customer experience.

[1380] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1381] In this invention, the server includes means for extracting text information from an image acquired by a camera using a character recognition engine, means for analyzing the extracted text information and surrounding background information to identify the next suggested task to be performed, means for detecting the user's facial expression and tone of voice to recognize emotions, and means for sorting the suggested tasks in order of priority based on the emotional information, thereby making it possible to suggest and execute appropriate tasks based on the emotional states of store staff and customers.

[1382] A "character recognition engine" is a device or program that analyzes and extracts character information from images acquired by a camera.

[1383] "Surrounding background information" is a general term for accompanying data and environmental information related to text information.

[1384] A "proposed task" is a proposal for the next work to be done based on the analyzed and parsed information.

[1385] "User notification means" refers to the means by which the proposed task is communicated to the user, displayed as a notification or alert.

[1386] A "task executor" is a processing mechanism for executing a suggested task based on user input.

[1387] "Facial expression detection" is a process that uses a camera to analyze a user's facial expressions and recognize their emotions.

[1388] "Tone of voice detection" is a process that uses a microphone to analyze the tone of a user's voice and recognize their emotions.

[1389] "Emotion recognition" is the process of identifying a user's emotions from facial expressions and tone of voice.

[1390] The "means for sorting by priority" is a processing function that rearranges the order of proposed tasks based on their importance and ease of execution.

[1391] "Multi-modality artificial intelligence" is an artificial intelligence system that combines and analyzes multiple data formats (for example, text information and background information) to provide users with appropriate information.

[1392] This invention uses a "smart store assistant" system in a physical store to efficiently and effectively suggest tasks to store staff and customers and assist them in carrying them out. A specific embodiment of this system will be described below.

[1393] First, the device uses a camera to scan the barcode or label of a product in a physical store. This device can be a smartphone or smart glasses. The camera is activated and the captured image is temporarily stored on the device. The image is then sent to a server for pre-processing. This pre-processing involves removing noise from the image and adjusting its resolution to improve the accuracy of character recognition by the OCR engine.

[1394] Next, the server inputs the preprocessed image data into an OCR engine, such as Tesseract OCR or Google Cloud Vision API. This extracts the text information from the image and records it as text data. At this stage, information such as the product name, price, and expiration date are obtained.

[1395] The extracted text information is input into a multi-modality AI system. The server analyzes the text information in combination with the surrounding contextual information to identify tasks to be proposed. For example, this could include suggesting restocking of low-stock items or proposing promotions for specific products.

[1396] The device detects the user's facial expressions and tone of voice through a camera and microphone, and analyzes them with an emotion recognition engine. The recognized emotion information is sent to the server and reflected in task suggestions. For example, if the user is tired, the device will prioritize simple tasks to reduce the user's burden.

[1397] Finally, the server sends the prioritized task list to the device. The device notifies the user of the task list through a pop-up window or an alert. When the user selects one of the suggested tasks, the task is automatically executed. For example, it may send a stock replenishment request or start a sales promotion campaign. Once the task is completed, the device feeds back the results to the server. The server stores this in a database and uses it to improve the accuracy of future suggestions.

[1398] For example, here's a prompt to notify you when an item is low in stock:

[1399] Scan the receipt image.

[1400] Important Task: This item is low on stock and needs to be restocked. Would you like to submit a restock request?

[1401] This system will reduce the burden on store staff in physical stores, enable them to carry out their work more efficiently, and improve customer satisfaction.

[1402] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1403] Step 1:

[1404] The device uses a camera to scan products or labels in a physical store. The user picks up the device (smartphone or smart glasses) and points the camera at the target product to capture an image. In this step, the camera is activated and the captured image is temporarily saved in the device. The image is then sent to the server. The input is the image of the product or label, and the output is the image data sent to the server.

[1405] Step 2:

[1406] The server preprocesses the received image data. Preprocessing includes noise removal and resolution adjustment. Specifically, the server converts the image to grayscale and applies Gaussian blur to remove noise. It also adjusts the resolution appropriately to improve the accuracy of character recognition. The input is the received raw image data, and the output is preprocessed, high-quality image data.

[1407] Step 3:

[1408] The server inputs the preprocessed image data into the OCR engine. The OCR engine (for example, Tesseract OCR or Google Cloud Vision API) analyzes the character information in the image and extracts it as text data. In this step, the server calls the OCR engine to perform the analysis and saves the obtained text data on the server. The input is the preprocessed image data, and the output is text data containing character information.

[1409] Step 4:

[1410] The server inputs the extracted text information and surrounding background information into a multi-modality AI for analysis. This identifies suggested tasks that are highly relevant to the user. The server then integrates the text information and background information and analyzes it using an AI model. As a result, a list of suggested tasks is generated. The input is text information and background information, and the output is a list of suggested tasks.

[1411] Step 5:

[1412] The device detects the user's facial expressions and tone of voice through a camera and microphone, and analyzes them using an emotion recognition engine. When the user operates the device, the camera captures the user's facial expressions and the microphone records the user's tone of voice. This data is sent from the device to a server and analyzed by the emotion recognition engine. The input is facial expression and voice data, and the output is recognized emotional information.

[1413] Step 6:

[1414] The server re-evaluates the proposed task list based on the emotion information and sorts it in order of priority. The server receives the emotion recognition results and adjusts the task priority according to the user's state. For example, if the user is tired, it prioritizes easier tasks. The input is the proposed task list and emotion information, and the output is a task list sorted in order of priority.

[1415] Step 7:

[1416] The terminal notifies the user of a priority-ordered task list sent from the server. The terminal displays the task list in a pop-up window or alert format, allowing the user to select from the list. When the user selects a specific task, that task is executed. The input is a task list sorted by priority, and the output is a notification to the user and the execution of the task.

[1417] Step 8:

[1418] After completing a task, the device sends the result (success, failure, feedback) to the server. The server stores the received feedback in a database and uses it as training data for the AI ​​model to improve the accuracy of the next task proposal. The input is the task execution result and feedback, and the output is the feedback information stored as learning data.

[1419] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1420] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1421] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1422] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1423] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1424] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1425] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1426] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1427] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1428] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1429] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1430] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1431] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1432] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1433] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1434] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1435] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1436] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1437] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1438] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1439] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1440] The following is further disclosed regarding the above embodiment.

[1441] (Claim 1)

[1442] means for extracting character information from an image acquired by a camera using a character recognition engine;

[1443] A means for analyzing the extracted character information and surrounding background information to identify a proposed task to be performed next;

[1444] a means of notifying the user of the proposed task;

[1445] A system including a means for performing suggested tasks based on user input.

[1446] (Claim 2)

[1447] 10. The system of claim 1, further comprising means for using multi-modality artificial intelligence to analyze surrounding context information.

[1448] (Claim 3)

[1449] 2. The system according to claim 1, further comprising means for recording the results of the execution of the proposed task and utilizing the results as learning data to improve the accuracy of the next proposal.

[1450] "Example 1"

[1451] (Claim 1)

[1452] means for temporarily storing images acquired from the camera in an internal storage device;

[1453] means for transmitting the image from the internal storage device to a server for noise reduction and resolution optimization;

[1454] means for using a character recognition engine to extract text information from the denoised and resolution optimised image;

[1455] A means for analyzing the extracted character information and surrounding background information using multi-modality artificial intelligence and identifying a proposed task to be performed next;

[1456] a notification means for notifying the user of the proposed task;

[1457] means for performing suggested tasks based on user input; and

[1458] A means for sending feedback after the task is completed to a server, storing the feedback in a database, and using it as learning data;

[1459] A system including:

[1460] (Claim 2)

[1461] 10. The system of claim 1, further comprising means for analyzing the extracted textual information and surrounding contextual information using multi-modality artificial intelligence.

[1462] (Claim 3)

[1463] 2. The system of claim 1, further comprising means for recording the results of the proposed tasks and utilizing the results as learning data to improve the accuracy of the next proposal.

[1464] "Application Example 1"

[1465] (Claim 1)

[1466] means for extracting character information from an image acquired by a camera using a character recognition engine;

[1467] A means for analyzing the extracted character information and surrounding background information to identify a proposed task to be performed next;

[1468] a means of notifying the user of the proposed task;

[1469] means for performing suggested tasks based on user input; and

[1470] means for performing pre-processing of the image by denoising and optimizing the resolution;

[1471] a means for performing cloud-based analytics;

[1472] A system including means for suggesting tasks such as comparing product prices and displaying reviews.

[1473] (Claim 2)

[1474] 10. The system of claim 1, further comprising means for using multi-modality artificial intelligence to analyze surrounding context information.

[1475] (Claim 3)

[1476] 2. The system according to claim 1, further comprising means for recording the results of the execution of the proposed task and utilizing the results as learning data to improve the accuracy of the next proposal.

[1477] "Example 2: Combining Emotion Engines"

[1478] (Claim 1)

[1479] means for activating the camera to acquire images;

[1480] means for pre-processing the acquired images to remove noise and optimize resolution;

[1481] using a character recognition engine to extract character information from the preprocessed image;

[1482] A means for analyzing the extracted character information and surrounding background information to identify a proposed task to be performed next;

[1483] a means for using an emotion engine to recognize the emotion of a user;

[1484] a means for re-evaluating the task list based on the recognized emotional information;

[1485] a means of notifying the user of the proposed task;

[1486] means for performing suggested tasks based on user input; and

[1487] The system includes a means for recording execution results and emotional information and evaluating them to improve the accuracy of suggestions next time.

[1488] (Claim 2)

[1489] 10. The system of claim 1, further comprising means for using multi-modality artificial intelligence to analyze surrounding context information.

[1490] (Claim 3)

[1491] 2. The system according to claim 1, further comprising means for recording the results of the execution of the proposed task and utilizing the results as learning data to improve the accuracy of the next proposal.

[1492] "Application example 2 when combining emotion engines"

[1493] (Claim 1)

[1494] means for extracting character information from an image acquired by a camera using a character recognition engine;

[1495] A means for analyzing the extracted character information and surrounding background information to identify a proposed task to be performed next;

[1496] a means of notifying the user of the proposed task;

[1497] means for performing suggested tasks based on user input; and

[1498] A means of recognizing emotions by detecting the user's facial expressions and tone of voice;

[1499] The system includes a means for sorting proposed tasks in order of priority based on emotional information.

[1500] (Claim 2)

[1501] 10. The system of claim 1, further comprising means for using multi-modality artificial intelligence to analyze surrounding context information.

[1502] (Claim 3)

[1503] 2. The system according to claim 1, further comprising means for recording the results of the execution of the proposed task and utilizing the results as learning data to improve the accuracy of the next proposal. [Explanation of symbols]

[1504] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for extracting character information from an image acquired by a camera using a character recognition engine; A means for analyzing the extracted character information and surrounding background information to identify a proposed task to be performed next; a means of notifying the user of the proposed task; A system including a means for performing suggested tasks based on user input.

2. 10. The system of claim 1, further comprising means for using multi-modality artificial intelligence to analyze surrounding context information.

3. The system according to claim 1, further comprising means for recording the results of the execution of the proposed task and utilizing the results as learning data to improve the accuracy of the next proposal.

Images