system

Abstract

We provide the system. [Solution] An image acquisition means that receives image data from an image acquisition device, Image processing means that preprocesses the aforementioned image data and extracts a specific part of the object, A character recognition means that analyzes character information from the image of the extracted object and converts it into a data format, A data structuring means that organizes the character information converted to the aforementioned data format as structured data and outputs it in tabular format, A system including a communication means for transmitting the aforementioned tabular data to a remote device.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot's character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Paper-based materials and manual data entry are still commonly performed in many companies, which brings problems of efficiency and accuracy in data collection and management. As a result, manual errors and time waste occur, which are factors damaging the competitiveness of companies. Also, it is difficult to handle various data formats, and there are restrictions on data integration and reuse. To solve these problems, there is a demand for a system that can perform efficient and accurate data input and management.

Means for Solving the Problems

[0005] To solve this problem, the present invention provides a system comprising: an image acquisition means for receiving image data from an image acquisition device; an image processing means for performing preprocessing on the received image data to extract specific parts of an object; and a character recognition means for analyzing character information from the extracted image and converting it into a data format. Furthermore, the system includes a data structuring means for organizing the converted character information as structured data and outputting it in a tabular format, and this data can be transmitted to a remote device via a communication means. This makes it possible to efficiently and accurately convert data from paper media into a digital format and make it usable in various data formats.

[0006] "Image acquisition means" refers to the function or device that receives image data from a device that acquires images.

[0007] "Image processing means" refers to a function or device that performs preprocessing on received image data and executes processing to extract specific information.

[0008] "Character recognition means" refers to a function or device that analyzes a specific portion extracted from an image and converts the character information into a data format.

[0009] "Data structuring means" refers to functions or devices that organize analyzed character information, convert it into a specific format or structure, and output it in tabular format.

[0010] "Communication means" refers to functions or devices for transmitting or receiving processed data to or from another device or apparatus. [Brief explanation of the drawing]

[0011] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0012] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0013] First, let's explain the terminology used in the following explanation.

[0014] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), and the like.

[0015] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0016] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs, various parameters, and the like. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disk (e.g., hard disk), or magnetic tape, and the like.

[0017] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor, an antenna, and the like. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0018] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0019] [First Embodiment]

[0020] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0021] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0022] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0023] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0024] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0025] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0026] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0027] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0028] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0029] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0030] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0031] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0032] This invention provides a system that allows users to convert information from paper documents into digital data. First, the user uses a smartphone or tablet as an image acquisition means to take pictures of the paper documents, such as receipts or invoices. The terminal collects this image data and transmits it to a server.

[0033] The server uses image processing equipment to preprocess the received image. Specifically, it removes noise from the image and adjusts the contrast to improve reading accuracy. Then, it automatically crops the important parts of the receipt and performs OCR processing using character recognition equipment. This extracts text data from the image.

[0034] The extracted text data is organized into categories such as item name, quantity, and price using data structuring methods on the server. This data is converted into a spreadsheet format and stored in a way that makes it usable with other business tools and databases. This spreadsheet data is then transmitted to the terminal using communication methods and displayed to the user.

[0035] As a concrete example, if this system is implemented in the retail industry for expense management, users can take a picture of the paper receipt they receive at the store, instantly digitizing the purchase information and importing it into the accounting system. This process is expected to improve the accuracy of data entry and reduce manual work, thereby increasing operational efficiency.

[0036] In this system, users, terminals, and servers work together to ensure that information is accurately digitized and managed, thereby facilitating a smooth transition from paper documents to digital data.

[0037] The following describes the processing flow.

[0038] Step 1:

[0039] Users use their smartphones to launch a dedicated app and take photos of receipts and invoices with their camera. The captured images are reviewed within the app, and if there are no problems, the images are temporarily saved on the device.

[0040] Step 2:

[0041] The terminal performs basic preprocessing on the stored image data. This preprocessing includes image compression, format conversion, and encryption for communication. The processed images are then sent to the server.

[0042] Step 3:

[0043] The server analyzes the image data received from the terminal. Image processing tools are used to remove noise from the image and adjust the resolution to improve reading accuracy. Distortion correction and binning are also performed during this process.

[0044] Step 4:

[0045] The server applies OCR (Optical Character Recognition) technology to the image-processed data. Here, the character recognition means extracts the strings written on the receipt as text data and classifies them according to the purchased items.

[0046] Step 5:

[0047] The server structures the acquired text data using data structuring methods. Product names and prices are organized in a tabular format and converted to a format suitable for spreadsheets. Automatic calculations are also performed for items requiring calculations (e.g., total amount).

[0048] Step 6:

[0049] The server uses communication methods to send structured data to the terminal. The data is exported in a format that suits the user's needs, such as JSON or CSV.

[0050] Step 7:

[0051] The terminal displays the spreadsheet data received from the server and allows the user to review it. The user can then edit, save, or integrate this data with other business systems as needed.

[0052] (Example 1)

[0053] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0054] There is a need to quickly and accurately convert information recorded on paper into digital data for management and utilization. Traditional manual data entry is time-consuming and prone to errors, making it a challenge to improve efficiency and accuracy.

[0055] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0056] In this invention, the server includes data acquisition means for receiving visual data from an image acquisition device, visual processing means for pre-processing the visual data and extracting important parts of the information medium, and text recognition means for automatically analyzing text information from the extracted visuals of the information medium and converting it into electronic data format. This enables rapid and accurate conversion from paper media to digital data.

[0057] An "image acquisition device" is a hardware device used to acquire information from paper documents as digital data.

[0058] "Visual data" refers to information acquired by an image acquisition device, represented in digital format.

[0059] "Data acquisition means" refers to a function for receiving visual data transmitted from an image acquisition device.

[0060] A "visual processing means" is a function that preprocesses visual data to extract only the necessary information.

[0061] An "information medium" refers to an object or information on which text or images are recorded.

[0062] "Important parts" refer to specific sections of information recorded on an information medium that require digital conversion.

[0063] A "text recognition means" is a function that analyzes text information from visual data and converts it into an electronic data format.

[0064] An "electronic data format" is a format used to digitize information.

[0065] A "data organization method" is a function for organizing electronic data into a specific format and providing it as usable data.

[0066] A "data transmission means" is a communication function for transferring organized data to other devices or systems.

[0067] The system for carrying out the present invention is configured by combining multiple hardware and software components to efficiently collect, process, and digitize visual data.

[0068] Users acquire images of paper documents using smartphones, tablets, or other image acquisition devices. This allows users to transmit the information from the paper documents as digital data to a server. Since this transmission utilizes the device's communication capabilities, Wi-Fi and mobile networks can be used.

[0069] The server first preprocesses the received visual data using image processing software. Software such as OpenCV or ImageMagick is used for this preprocessing. Preprocessing removes noise and adjusts contrast to enable more accurate recognition of text information. Next, the server extracts text data from the image using OCR (Optical Character Recognition) software such as Tesseract.

[0070] Furthermore, the extracted text data is structured using a database management system. Specifically, it is organized into items such as product name, quantity, and price, and then converted and saved in spreadsheet format. Database software such as MySQL® or PostgreSQL is used for this purpose.

[0071] The server ultimately sends the structured spreadsheet data to the device. On the device, this data can be displayed in applications such as Excel or Google Sheets, allowing the user to view and manipulate the data.

[0072] As a concrete example, imagine a user taking a picture of a paper receipt at a retail store and digitizing that information for expense management. By using this system, users can efficiently improve the accuracy of data entry and streamline their business processes.

[0073] A possible prompt for the generative AI model would be, "Please tell me how to convert paper receipts into digital data and manage them efficiently."

[0074] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0075] Step 1:

[0076] The user takes a photograph of a paper document using a smartphone or tablet, which is an image acquisition device. This image data is then stored on the device. The input is paper, and the output is digital image data. Specifically, the user launches a camera app and takes a picture of a receipt or invoice.

[0077] Step 2:

[0078] The device sends stored image data to the server. The input is digital image data stored on the device, and the output is the data sent to the server. Here, the device uses Wi-Fi or mobile data communication to send data to the server over the network.

[0079] Step 3:

[0080] The server performs preprocessing on the received image data. Specifically, it performs noise reduction and contrast adjustment. The input is the transmitted image data, and the output is the image data with noise reduction and contrast adjustment. Image processing software such as ImageMagick and OpenCV is used to perform processing to improve visibility.

[0081] Step 4:

[0082] The server extracts text information from pre-processed image data. Here, character recognition is performed using OCR technology. The input is pre-processed image data, and the output is text data. This involves the specific operation of converting the text portion of the image into text format using OCR software such as Tesseract.

[0083] Step 5:

[0084] The server structures the extracted text data. It organizes the text data into categories such as item name, quantity, and price, and converts it into a spreadsheet format. The input is unorganized text data, and the output is structured spreadsheet data. A database management system such as MySQL or PostgreSQL is used to appropriately classify and store the data.

[0085] Step 6:

[0086] The server sends structured data to the terminal, and the user views and reviews this data. The input is structured spreadsheet data, and the output is data displayed on the terminal. The data can be viewed and used by the user through applications such as Excel or Google Sheets.

[0087] (Application Example 1)

[0088] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0089] Current online shopping return procedures are cumbersome and time-consuming for buyers. Furthermore, manually entering receipt information increases the likelihood of human error. Therefore, it is necessary to simplify the return process and automate it using digital information.

[0090] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0091] In this invention, the server includes an image acquisition means for receiving image data from an image acquisition device, an image processing means for automatically extracting important information, a character recognition means for converting character information into a digital data format, a data structuring means for linking purchase information and outputting data in tab format, and a communication means for automating the return process. This allows users to perform the return procedure smoothly and significantly reduce the time and effort required.

[0092] "Image acquisition means" refers to the technical elements for receiving image data from a device.

[0093] "Image processing means" refers to technical elements that perform preprocessing on image data and automatically extract important information.

[0094] "Character recognition means" refers to the technical elements used to analyze character information from extracted information and convert it into digital data format.

[0095] "Data structuring means" refers to technical elements for linking text information with purchase information and outputting it in a structured tab format.

[0096] "Communication means" refers to the technical elements used to transmit tab-formatted data to a remote computing device and automate the return processing.

[0097] The system designed to realize this application seamlessly and efficiently supports users in the return process using a smartphone app.

[0098] The user takes a picture of the receipt for the item they want to return using their smartphone camera. The captured image is optimized by an application on the device and sent to the server. The server uses an image processing library (e.g., OpenCV) to remove noise from the received image data, crops the necessary areas, and extracts important information. Next, an OCR engine (e.g., Tesseract OCR) is used to analyze the text information from the image and convert it into digital data.

[0099] The converted digital data is linked to purchase information and stored in a database. A data processing library (e.g., Pandas) is used to structure the data. It is then organized as tabbed data and transmitted to a remote computing device via a REST API. Finally, the user can confirm that the return process proceeds automatically within the application.

[0100] As a concrete example of this system, consider a case where a user purchases clothing online but the size is incorrect. In this case, the user takes a picture of the receipt for the clothing and selects to proceed with the return process within the app. This entire process significantly simplifies the return process and improves the user experience. As an example of a prompt message to the generating AI model, instructions such as "I would like to return this item. I have taken a picture of the receipt, so please proceed with the return process" can be entered into the application.

[0101] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0102] Step 1:

[0103] The terminal takes a picture of the receipt for the item the user wishes to return using the smartphone's camera. The input is the image of the photographed receipt, and the output is the image data stored on the terminal. The user's photography operation is seamlessly handled, and a process to verify the image resolution is performed.

[0104] Step 2:

[0105] The terminal optimizes the captured image data and sends it to the server. The input is the image data obtained in step 1, and the output is the optimized image data sent to the server. Specifically, the terminal compresses the image, adjusts the resolution, and sends the data via a communication protocol.

[0106] Step 3:

[0107] The server performs image processing on the received image data. The input is the image data received by the server, and the output is the denoised and trimmed image data. The server uses OpenCV to perform denoising and trim important parts.

[0108] Step 4:

[0109] The server analyzes the characters from the cropped image and converts them into digital data. The input is the cropped image from step 3, and the output is digital data in the form of text. The server performs OCR processing using Tesseract OCR to obtain the character information.

[0110] Step 5:

[0111] The server structures the retrieved text information and links it to purchase information. The input is the text data obtained in step 4, and the output is structured tab-formatted data. The server uses the Pandas library to organize the data and stores the purchased item names and price information in the database.

[0112] Step 6:

[0113] The server sends structured data to a remote computing device using a communication method. The input is tab-formatted data generated in step 5, and the output is data processed by the remote system. The server sends data via a REST API and is configured to automatically handle return processing.

[0114] Step 7:

[0115] The user confirms the completion of the return process on their smartphone application. The input is the processing result displayed on the device, and the output is the user's confirmation that the return process is complete. The user checks the in-app notification and recognizes that the process has been completed smoothly.

[0116] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0117] This invention is characterized by providing a system that not only allows users to convert various types of information into digital data, but also analyzes users' emotions in real time. First, the user takes a picture of a receipt or document with a smart device equipped with image acquisition means, and temporarily stores the image data on the terminal. The terminal sends this image to a server, and the server analyzes the image using image processing means.

[0118] On the server, OCR technology is applied to the received image data to perform character recognition and extract information written on paper as text data. Subsequently, the purchased item information is organized into a spreadsheet format using a data structuring method. At this time, the emotion engine is activated and analyzes the user's emotional state using the image and audio data obtained from the user.

[0119] The analysis results from the emotion engine are reflected in the customization of data output. For example, if a user is experiencing stress, the system can simplify the interface and display designs and suggestions that reduce stress. It also has a mechanism to adjust the priority and display format of data presentation based on emotions, quickly providing the user with the information they need most.

[0120] For example, if a user scans documents under tight time pressure before a meeting, the emotion engine detects the time-pressured and stressed state, and the device provides highlights of important information and priority notifications. This allows the user to quickly and accurately obtain the necessary information.

[0121] By adding an emotion engine, this system transcends being merely a data conversion tool, providing support functions that enhance the user experience. This enables users to manage information efficiently and with less stress.

[0122] The following describes the processing flow.

[0123] Step 1:

[0124] Users use a device such as a smartphone or tablet to launch the application and take a picture of a paper receipt or document. After taking the picture, the app temporarily saves the image to the device. At this point, users can also use options such as taking a selfie or voice input for emotion analysis.

[0125] Step 2:

[0126] The device sends captured image data and additional data from the user to the server. Before transmission, the data is compressed and encrypted to ensure secure communication. Images and audio are also transmitted as emotional data.

[0127] Step 3:

[0128] The server performs image processing on the image data received from the terminal. Specifically, it removes noise and crops the data, extracting only the necessary parts. In parallel, the emotion engine begins analyzing the user's emotions. This analysis is based on the received selfie images and audio information.

[0129] Step 4:

[0130] The server uses OCR technology to extract text data from images. The recognized text information is classified into categories such as purchase information and numerical values. Taking into account the results of the emotion engine's analysis, the data is organized into a spreadsheet format using data structuring methods.

[0131] Step 5:

[0132] The emotion engine instructs the user interface (UI) to be customized based on the analysis results. When the user is relaxed, it provides information using the normal UI, while when the user is experiencing stress, it prioritizes displaying important information. It also dynamically adjusts the priority of data output according to the user's emotions.

[0133] Step 6:

[0134] The server sends organized spreadsheet data and customized UI information to the device. The data is output in the format specified by the user and can be downloaded as CSV or JSON as needed.

[0135] Step 7:

[0136] The device displays data received from the server and UI information to the user. The user can review this data and, if necessary, edit, save, and share it. Furthermore, the displayed information is adjusted based on sentiment analysis, improving the user's information access experience.

[0137] (Example 2)

[0138] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0139] The problem that this invention aims to solve is to reduce user stress and enable efficient information management by providing adaptive data presentation that corresponds to the user's emotional state during the conversion process to digital data.

[0140] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0141] In this invention, the server includes acquisition means for receiving image information, analysis means for preprocessing, recognition means for converting character information into a data format, adjustment means for customizing data based on emotional state, and transmission means for transmitting data. This enables the display and management of information corresponding to the user's emotional state.

[0142] "Acquisition means" refers to a function or device for receiving image information from an information processing device.

[0143] "Analysis means" refers to a function or device for preprocessing received image information and extracting specific target information.

[0144] "Recognition means" refers to a function or device for analyzing textual information from extracted images and converting it into a data format.

[0145] A "processing method" is a function or device for organizing converted character information as structured data and outputting it in tabular format.

[0146] "Adjustment means" refers to a function or device that adaptively customizes data output based on the results of an analysis of the user's emotional state.

[0147] "Transmission means" refers to a function or device for securely transmitting adjusted data to a remote information processing device.

[0148] This system enables users to effectively digitize information and manage it using sentiment analysis. Users first use a smart device with image acquisition capabilities to photograph documents or receipts. The image data is temporarily stored on the smart device and then transmitted to the server using a secure communication protocol (e.g., HTTPS).

[0149] The server preprocesses images using an image processing library (e.g., OpenCV) to remove noise and correct tilt. Next, it uses OCR technology (e.g., Tesseract OCR) to identify text information and extract text data from the images. This text data is then organized into a spreadsheet format using a data structuring library (e.g., Python's Pandas).

[0150] Furthermore, the server uses an emotion analysis engine to analyze the emotional state from image and audio data provided by the user. For example, by utilizing natural language processing tools (e.g., Google Cloud Natural Language API), the user's emotional state can be understood in real time. Based on the results of the emotion analysis, the user interface is adjusted, and information that matches the user's needs is prioritized. For example, important information can be concisely summarized and highlighted for users who are feeling stressed.

[0151] For example, if a user scans a document while pressed for time before a meeting, the system detects this state of urgency and highlights important information for immediate access. In this way, the user can quickly and accurately obtain the necessary information.

[0152] For example, entering a prompt such as, "I scanned the documents before the meeting. I'm busy, so please prioritize showing me the most important information," will allow the system to provide more relevant information.

[0153] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0154] Step 1:

[0155] The user uses a smart device to photograph receipts or documents. The input for this procedure is the physical paper documents themselves. When the user photographs them, they are output as image data and temporarily stored on the smart device.

[0156] Step 2:

[0157] The terminal sends image data stored on the smart device to the server. The input for this procedure is the image data stored on the terminal. The terminal sends the image data to the server, and the image data is transferred to the server as output using a secure communication protocol (e.g., HTTPS).

[0158] Step 3:

[0159] The server analyzes the image data it receives. The input to this process is the image data that arrives at the server. The server uses an image processing library (e.g., OpenCV) to perform preprocessing such as noise reduction and tilt correction, and outputs the preprocessed image data.

[0160] Step 4:

[0161] The server extracts text information from image data using OCR technology. The input for this procedure is pre-processed image data. The server applies OCR (e.g., Tesseract OCR) to extract the characters in the image as text data and outputs that text data.

[0162] Step 5:

[0163] The server organizes text data into a spreadsheet format. The input for this process is the text data extracted by the server in a previous step. The server uses a data organization library (e.g., Python's Pandas) to structure the text data and output it as a spreadsheet.

[0164] Step 6:

[0165] The server uses an emotion analysis engine to analyze the user's emotions. The user's image and audio data are used as input for the emotion analysis. The server utilizes natural language processing tools (e.g., Google Cloud Natural Language API) to analyze the user's emotional state and outputs the results.

[0166] Step 7:

[0167] The server customizes the data output based on the analysis results. The input for this procedure is the analysis results from the sentiment analysis engine. Based on this, the server adjusts the interface design and how information is presented, and outputs customized data adapted to the user.

[0168] (Application Example 2)

[0169] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0170] Many modern consumers seek ways to manage information about their purchased products and optimize their purchasing experience according to their individual emotional states. However, conventional information management systems struggle to consider users' emotional states and fail to adequately improve the consumer experience. In this context, there is a need to develop systems that enable flexible and appropriate information presentation based on users' emotions.

[0171] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0172] In this invention, the server includes means for receiving image data from an image acquisition device, processing means for pre-processing the image data and extracting specific parts of an object, and recognition means for analyzing textual information from the extracted image of the object and converting it into an information format. This makes it possible to easily organize information about products purchased by consumers, and further analyze the user's emotional state based on that information, enabling the presentation of information tailored to their emotions.

[0173] A "device" is a piece of equipment used to acquire or process information.

[0174] "Means" refer to methods or components used to achieve a specific function or purpose.

[0175] "Image data" refers to visual information represented in electronic format.

[0176] "Preprocessing" refers to preparatory work performed before data analysis, and is carried out to improve the quality of the data.

[0177] "Subject matter" refers to the item or data that is the subject of analysis or manipulation.

[0178] "Extraction" is the act of taking useful information or elements from the original data.

[0179] "Analysis" is the process of examining data and phenomena in detail to clarify their structure and meaning.

[0180] "Information format" refers to the method of presenting data in a way that is easy to process visually or mechanically.

[0181] "Structured information" refers to data that is organized in an orderly manner and follows a specific format or pattern.

[0182] "Presentation" means the act of making information public or displaying it in a visual or other form.

[0183] "Distant" refers to a location or point that is physically far away.

[0184] The system implementing this invention mainly consists of an image acquisition device, a server, and a user terminal. After receiving image data from the user, the server performs data preprocessing and character recognition using OpenCV and Tesseract. Furthermore, it analyzes the user's emotional state from images and audio using TENSORFLOW® and Keras. The analyzed emotional information is presented to the user in a structured format, and the most appropriate product information and suggestions are instantly provided. Through this series of processes, the user can obtain a personalized shopping experience via a dedicated smartphone application.

[0185] For example, suppose a user scans a receipt after shopping at a store. The server uses OCR technology to convert the receipt information into text data, and then uses TensorFlow to analyze the user's emotions from their facial image based on that information. If the system determines that the user is tired, it will then present promotions for relaxing products and services in the app based on that information.

[0186] The application can generate prompt messages using a generative AI model. An example of a prompt message that would allow this application to function effectively is: "The user is in a state of high stress. Generate a prompt suggesting a product or service that would be effective in promoting relaxation." This example prompt message provides the foundation for the system to take actions in response to the user's emotional state.

[0187] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0188] Step 1:

[0189] The user launches the smartphone app and takes a picture of the receipt after making a purchase. The input is the image data of the photographed receipt. This image data is temporarily stored within the app.

[0190] Step 2:

[0191] The device sends image data acquired through the app to the server. The data sent is the raw data of the receipt image. The server receives the image data.

[0192] Step 3:

[0193] The server uses OpenCV to preprocess image data. The input is receipt image data, and the output is a clear image with noise reduction. This processing improves the image quality.

[0194] Step 4:

[0195] The server uses Tesseract with OCR technology to recognize characters from pre-processed images and extract text data. The input is pre-processed image data, and the output is extracted text data. The character information is digitized.

[0196] Step 5:

[0197] The server receives the text data obtained from character recognition and converts it into an informational format. This output is in a structured data format, such as a spreadsheet. The data is organized into a format that can be viewed at a glance.

[0198] Step 6:

[0199] The server uses TensorFlow and Keras to analyze emotions from the user's facial image or voice data. The input is the user's facial image or voice data, and the output is the emotion analysis result. The user's emotional state is identified.

[0200] Step 7:

[0201] The server uses a generative AI model based on analyzed sentiment data to provide the user with the most relevant information. This is done according to guidelines in the generated prompt text. The user is then provided with the most appropriate product information and promotions.

[0202] Step 8:

[0203] The device receives information sent from the server and displays it on the smartphone's screen. The output consists of information and campaign information customized for the user. Users can enjoy the most suitable offers based on their purchase history and reward information.

[0204] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0205] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0206] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0207] [Second Embodiment]

[0208] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0209] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0210] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0211] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0212] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0213] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0214] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0215] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0216] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0217] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0218] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0219] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0220] This invention provides a system that allows users to convert information from paper documents into digital data. First, the user uses a smartphone or tablet as an image acquisition means to take pictures of the paper documents, such as receipts or invoices. The terminal collects this image data and transmits it to a server.

[0221] The server uses image processing equipment to preprocess the received image. Specifically, it removes noise from the image and adjusts the contrast to improve reading accuracy. Then, it automatically crops the important parts of the receipt and performs OCR processing using character recognition equipment. This extracts text data from the image.

[0222] The extracted text data is organized into categories such as item name, quantity, and price using data structuring methods on the server. This data is converted into a spreadsheet format and stored in a way that makes it usable with other business tools and databases. This spreadsheet data is then transmitted to the terminal using communication methods and displayed to the user.

[0223] As a concrete example, if this system is implemented in the retail industry for expense management, users can take a picture of the paper receipt they receive at the store, instantly digitizing the purchase information and importing it into the accounting system. This process is expected to improve the accuracy of data entry and reduce manual work, thereby increasing operational efficiency.

[0224] In this system, users, terminals, and servers work together to ensure that information is accurately digitized and managed, thereby facilitating a smooth transition from paper documents to digital data.

[0225] The following describes the processing flow.

[0226] Step 1:

[0227] Users use their smartphones to launch a dedicated app and take photos of receipts and invoices with their camera. The captured images are reviewed within the app, and if there are no problems, the images are temporarily saved on the device.

[0228] Step 2:

[0229] The terminal performs basic preprocessing on the stored image data. This preprocessing includes image compression, format conversion, and encryption for communication. The processed images are then sent to the server.

[0230] Step 3:

[0231] The server analyzes the image data received from the terminal. Image processing tools are used to remove noise from the image and adjust the resolution to improve reading accuracy. Distortion correction and binning are also performed during this process.

[0232] Step 4:

[0233] The server applies OCR (Optical Character Recognition) technology to the image-processed data. Here, the character recognition means extracts the strings written on the receipt as text data and classifies them according to the purchased items.

[0234] Step 5:

[0235] The server structures the acquired text data using data structuring methods. Product names and prices are organized in a tabular format and converted to a format suitable for spreadsheets. Automatic calculations are also performed for items requiring calculations (e.g., total amount).

[0236] Step 6:

[0237] The server uses communication methods to send structured data to the terminal. The data is exported in a format that suits the user's needs, such as JSON or CSV.

[0238] Step 7:

[0239] The terminal displays the spreadsheet data received from the server and allows the user to review it. The user can then edit, save, or integrate this data with other business systems as needed.

[0240] (Example 1)

[0241] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0242] There is a need to quickly and accurately convert information recorded on paper into digital data for management and utilization. Traditional manual data entry is time-consuming and prone to errors, making it a challenge to improve efficiency and accuracy.

[0243] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0244] In this invention, the server includes data acquisition means for receiving visual data from an image acquisition device, visual processing means for pre-processing the visual data and extracting important parts of the information medium, and text recognition means for automatically analyzing text information from the extracted visuals of the information medium and converting it into electronic data format. This enables rapid and accurate conversion from paper media to digital data.

[0245] An "image acquisition device" is a hardware device used to acquire information from paper documents as digital data.

[0246] "Visual data" refers to information acquired by an image acquisition device, represented in digital format.

[0247] "Data acquisition means" refers to a function for receiving visual data transmitted from an image acquisition device.

[0248] A "visual processing means" is a function that preprocesses visual data to extract only the necessary information.

[0249] An "information medium" refers to an object or information on which text or images are recorded.

[0250] "Important parts" refer to specific sections of information recorded on an information medium that require digital conversion.

[0251] A "text recognition means" is a function that analyzes text information from visual data and converts it into an electronic data format.

[0252] An "electronic data format" is a format used to digitize information.

[0253] A "data organization method" is a function for organizing electronic data into a specific format and providing it as usable data.

[0254] A "data transmission means" is a communication function for transferring organized data to other devices or systems.

[0255] The system for carrying out the present invention is configured by combining multiple hardware and software components to efficiently collect, process, and digitize visual data.

[0256] Users acquire images of paper documents using smartphones, tablets, or other image acquisition devices. This allows users to transmit the information from the paper documents as digital data to a server. Since this transmission utilizes the device's communication capabilities, Wi-Fi and mobile networks can be used.

[0257] The server first preprocesses the received visual data using image processing software. Software such as OpenCV or ImageMagick is used for this preprocessing. Preprocessing removes noise and adjusts contrast to enable more accurate recognition of text information. Next, the server extracts text data from the image using OCR (Optical Character Recognition) software such as Tesseract.

[0258] Furthermore, the extracted text data is structured using a database management system. Specifically, it is organized into items such as item name, quantity, and price, and then converted and saved in spreadsheet format. Database software such as MySQL or PostgreSQL is used for this purpose.

[0259] The server ultimately sends the structured spreadsheet data to the device. On the device, this data is displayed in an application such as Excel or Google Sheets, allowing the user to view and manipulate the data.

[0260] As a concrete example, imagine a user taking a picture of a paper receipt at a retail store and digitizing that information for expense management. By using this system, users can efficiently improve the accuracy of data entry and streamline their business processes.

[0261] A possible prompt for the generative AI model would be, "Please tell me how to convert paper receipts into digital data and manage them efficiently."

[0262] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0263] Step 1:

[0264] The user takes a photograph of a paper document using a smartphone or tablet, which is an image acquisition device. This image data is then stored on the device. The input is paper, and the output is digital image data. Specifically, the user launches a camera app and takes a picture of a receipt or invoice.

[0265] Step 2:

[0266] The device sends stored image data to the server. The input is digital image data stored on the device, and the output is the data sent to the server. Here, the device uses Wi-Fi or mobile data communication to send data to the server over the network.

[0267] Step 3:

[0268] The server performs preprocessing on the received image data. Specifically, it performs noise reduction and contrast adjustment. The input is the transmitted image data, and the output is the image data with noise reduction and contrast adjustment. Image processing software such as ImageMagick and OpenCV is used to perform processing to improve visibility.

[0269] Step 4:

[0270] The server extracts text information from pre-processed image data. Here, character recognition is performed using OCR technology. The input is pre-processed image data, and the output is text data. This involves the specific operation of converting the text portion of the image into text format using OCR software such as Tesseract.

[0271] Step 5:

[0272] The server structures the extracted text data. It organizes the text data into categories such as item name, quantity, and price, and converts it into a spreadsheet format. The input is unorganized text data, and the output is structured spreadsheet data. A database management system such as MySQL or PostgreSQL is used to appropriately classify and store the data.

[0273] Step 6:

[0274] The server sends structured data to the terminal, and the user views and reviews this data. The input is structured spreadsheet data, and the output is data displayed on the terminal. The data can be viewed and used by the user through applications such as Excel or Google Sheets.

[0275] (Application Example 1)

[0276] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0277] Current online shopping return procedures are cumbersome and time-consuming for buyers. Furthermore, manually entering receipt information increases the likelihood of human error. Therefore, it is necessary to simplify the return process and automate it using digital information.

[0278] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0279] In this invention, the server includes an image acquisition means for receiving image data from an image acquisition device, an image processing means for automatically extracting important information, a character recognition means for converting character information into a digital data format, a data structuring means for linking purchase information and outputting data in tab format, and a communication means for automating the return process. This allows users to perform the return procedure smoothly and significantly reduce the time and effort required.

[0280] "Image acquisition means" refers to the technical elements for receiving image data from a device.

[0281] "Image processing means" refers to technical elements that perform preprocessing on image data and automatically extract important information.

[0282] The "character recognition means" is a technical element for analyzing character information from the extracted information and converting it into a digital data format.

[0283] The "data structuring means" is a technical element for linking character information with purchase information, structuring it in tabular form, and outputting it.

[0284] The "communication means" is a technical element for transmitting tabular data to a computing device at a remote location and automating the return process.

[0285] The system for realizing this application example seamlessly and efficiently supports the return procedure that the user performs using a smartphone app.

[0286] The user takes a photo of the receipt of the product to be returned with the camera of the smartphone. The captured image is optimized by the application on the terminal and sent to the server. The server uses an image processing library (e.g., OpenCV) to remove noise from the received image data, trim the necessary area, and extract important information. Next, character information is analyzed from the image using an OCR engine (e.g., Tesseract OCR) and converted into digital data.

[0287] The converted digital data is linked with the purchase information and stored in the database. At this time, a data processing library (e.g., Pandas) is used for data structuring. Then, it is organized as tabular data and sent to a computing device at a remote location by the communication means via the REST API. Finally, the user can confirm that the return procedure automatically progresses on the application.

[0288] As a concrete example of this system, consider a case where a user purchases clothing online but the size is incorrect. In this case, the user takes a picture of the receipt for the clothing and selects to proceed with the return process within the app. This entire process significantly simplifies the return process and improves the user experience. As an example of a prompt message to the generating AI model, instructions such as "I would like to return this item. I have taken a picture of the receipt, so please proceed with the return process" can be entered into the application.

[0289] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0290] Step 1:

[0291] The terminal takes a picture of the receipt for the item the user wishes to return using the smartphone's camera. The input is the image of the photographed receipt, and the output is the image data stored on the terminal. The user's photography operation is seamlessly handled, and a process to verify the image resolution is performed.

[0292] Step 2:

[0293] The terminal optimizes the captured image data and sends it to the server. The input is the image data obtained in step 1, and the output is the optimized image data sent to the server. Specifically, the terminal compresses the image, adjusts the resolution, and sends the data via a communication protocol.

[0294] Step 3:

[0295] The server performs image processing on the received image data. The input is the image data received by the server, and the output is the denoised and trimmed image data. The server uses OpenCV to perform denoising and trim important parts.

[0296] Step 4:

[0297] The server analyzes the characters from the cropped image and converts them into digital data. The input is the cropped image from step 3, and the output is digital data in the form of text. The server performs OCR processing using Tesseract OCR to obtain the character information.

[0298] Step 5:

[0299] The server structures the retrieved text information and links it to purchase information. The input is the text data obtained in step 4, and the output is structured tab-formatted data. The server uses the Pandas library to organize the data and stores the purchased item names and price information in the database.

[0300] Step 6:

[0301] The server sends structured data to a remote computing device using a communication method. The input is tab-formatted data generated in step 5, and the output is data processed by the remote system. The server sends data via a REST API and is configured to automatically handle return processing.

[0302] Step 7:

[0303] The user confirms the completion of the return process on their smartphone application. The input is the processing result displayed on the device, and the output is the user's confirmation that the return process is complete. The user checks the in-app notification and recognizes that the process has been completed smoothly.

[0304] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0305] The present invention is characterized by providing a system that not only allows users to convert various information into digital data but also analyzes users' emotions in real time. First, the user takes pictures of receipts and documents with a smart device equipped with image acquisition means and temporarily stores the image data in the terminal. The terminal then transmits this image to the server, and the server analyzes the image using image processing means.

[0306] At the server, the received image data is applied with OCR technology for character recognition to extract the information described on the paper medium as text data. Subsequently, the data structuring means arranges the purchased item information in a spreadsheet format. At this time, the emotion engine is activated, and using the image and voice data obtained from the user, it analyzes the user's emotional state.

[0307] The analysis results by the emotion engine are reflected in the customization of data output. For example, when the user is feeling stressed, the system can simplify the interface and display designs and suggestions that reduce stress. Also, it has a mechanism to adjust the priority and display format of data presentation based on emotions and quickly provide the information that the user most needs.

[0308] As a specific example, when the user scans documents in a situation where there is no time left before a meeting, the emotion engine detects the tense state of being pressed for time, and the terminal provides highlights of important information and priority notifications. Thereby, the user can obtain the necessary information quickly and accurately.

[0309] With the addition of the emotion engine, this system goes beyond a mere data conversion tool and provides a support function to improve the user experience. Thereby, it becomes possible for the user to manage information efficiently and with less stress.

[0310] The following explains the processing flow.

[0311] Step 1:

[0312] Users use a device such as a smartphone or tablet to launch the application and take a picture of a paper receipt or document. After taking the picture, the app temporarily saves the image to the device. At this point, users can also use options such as taking a selfie or voice input for emotion analysis.

[0313] Step 2:

[0314] The device sends captured image data and additional data from the user to the server. Before transmission, the data is compressed and encrypted to ensure secure communication. Images and audio are also transmitted as emotional data.

[0315] Step 3:

[0316] The server performs image processing on the image data received from the terminal. Specifically, it removes noise and crops the data, extracting only the necessary parts. In parallel, the emotion engine begins analyzing the user's emotions. This analysis is based on the received selfie images and audio information.

[0317] Step 4:

[0318] The server uses OCR technology to extract text data from images. The recognized text information is classified into categories such as purchase information and numerical values. Taking into account the results of the emotion engine's analysis, the data is organized into a spreadsheet format using data structuring methods.

[0319] Step 5:

[0320] The emotion engine instructs the user interface (UI) to be customized based on the analysis results. When the user is relaxed, it provides information using the normal UI, while when the user is experiencing stress, it prioritizes displaying important information. It also dynamically adjusts the priority of data output according to the user's emotions.

[0321] Step 6:

[0322] The server sends organized spreadsheet data and customized UI information to the device. The data is output in the format specified by the user and can be downloaded as CSV or JSON as needed.

[0323] Step 7:

[0324] The device displays data received from the server and UI information to the user. The user can review this data and, if necessary, edit, save, and share it. Furthermore, the displayed information is adjusted based on sentiment analysis, improving the user's information access experience.

[0325] (Example 2)

[0326] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0327] The problem that this invention aims to solve is to reduce user stress and enable efficient information management by providing adaptive data presentation that corresponds to the user's emotional state during the conversion process to digital data.

[0328] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0329] In this invention, the server includes acquisition means for receiving image information, analysis means for preprocessing, recognition means for converting character information into a data format, adjustment means for customizing data based on emotional state, and transmission means for transmitting data. This enables the display and management of information corresponding to the user's emotional state.

[0330] "Acquisition means" refers to a function or device for receiving image information from an information processing device.

[0331] "Analysis means" refers to a function or device for preprocessing received image information and extracting specific target information.

[0332] "Recognition means" refers to a function or device for analyzing textual information from extracted images and converting it into a data format.

[0333] A "processing method" is a function or device for organizing converted character information as structured data and outputting it in tabular format.

[0334] "Adjustment means" refers to a function or device that adaptively customizes data output based on the results of an analysis of the user's emotional state.

[0335] "Transmission means" refers to a function or device for securely transmitting adjusted data to a remote information processing device.

[0336] This system enables users to effectively digitize information and manage it using sentiment analysis. Users first use a smart device with image acquisition capabilities to photograph documents or receipts. The image data is temporarily stored on the smart device and then transmitted to the server using a secure communication protocol (e.g., HTTPS).

[0337] The server preprocesses images using an image processing library (e.g., OpenCV) to remove noise and correct tilt. Next, it uses OCR technology (e.g., Tesseract OCR) to identify text information and extract text data from the images. This text data is then organized into a spreadsheet format using a data structuring library (e.g., Python's Pandas).

[0338] Furthermore, the server uses an emotion analysis engine to analyze the emotional state from image and audio data provided by the user. For example, by utilizing natural language processing tools (e.g., Google Cloud Natural Language API), the user's emotional state can be understood in real time. Based on the results of the emotion analysis, the user interface is adjusted, and information that matches the user's needs is prioritized. For example, important information can be concisely summarized and highlighted for users who are feeling stressed.

[0339] For example, if a user scans a document while pressed for time before a meeting, the system detects this state of urgency and highlights important information for immediate access. In this way, the user can quickly and accurately obtain the necessary information.

[0340] For example, entering a prompt such as, "I scanned the documents before the meeting. I'm busy, so please prioritize showing me the most important information," will allow the system to provide more relevant information.

[0341] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0342] Step 1:

[0343] The user uses a smart device to photograph receipts or documents. The input for this procedure is the physical paper documents themselves. When the user photographs them, they are output as image data and temporarily stored on the smart device.

[0344] Step 2:

[0345] The terminal sends image data stored on the smart device to the server. The input for this procedure is the image data stored on the terminal. The terminal sends the image data to the server, and the image data is transferred to the server as output using a secure communication protocol (e.g., HTTPS).

[0346] Step 3:

[0347] The server analyzes the image data it receives. The input to this process is the image data that arrives at the server. The server uses an image processing library (e.g., OpenCV) to perform preprocessing such as noise reduction and tilt correction, and outputs the preprocessed image data.

[0348] Step 4:

[0349] The server extracts text information from image data using OCR technology. The input for this procedure is pre-processed image data. The server applies OCR (e.g., Tesseract OCR) to extract the characters in the image as text data and outputs that text data.

[0350] Step 5:

[0351] The server organizes text data into a spreadsheet format. The input for this process is the text data extracted by the server in a previous step. The server uses a data organization library (e.g., Python's Pandas) to structure the text data and output it as a spreadsheet.

[0352] Step 6:

[0353] The server uses an emotion analysis engine to analyze the user's emotions. The user's image and audio data are used as input for the emotion analysis. The server utilizes natural language processing tools (e.g., Google Cloud Natural Language API) to analyze the user's emotional state and outputs the results.

[0354] Step 7:

[0355] The server customizes the data output based on the analysis results. The input for this procedure is the analysis results from the sentiment analysis engine. Based on this, the server adjusts the interface design and how information is presented, and outputs customized data adapted to the user.

[0356] (Application Example 2)

[0357] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0358] Many modern consumers seek ways to manage information about their purchased products and optimize their purchasing experience according to their individual emotional states. However, conventional information management systems struggle to consider users' emotional states and fail to adequately improve the consumer experience. In this context, there is a need to develop systems that enable flexible and appropriate information presentation based on users' emotions.

[0359] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0360] In this invention, the server includes means for receiving image data from an image acquisition device, processing means for pre-processing the image data and extracting specific parts of an object, and recognition means for analyzing textual information from the extracted image of the object and converting it into an information format. This makes it possible to easily organize information about products purchased by consumers, and further analyze the user's emotional state based on that information, enabling the presentation of information tailored to their emotions.

[0361] A "device" is a piece of equipment used to acquire or process information.

[0362] "Means" refer to methods or components used to achieve a specific function or purpose.

[0363] "Image data" refers to visual information represented in electronic format.

[0364] "Preprocessing" refers to preparatory work performed before data analysis, and is carried out to improve the quality of the data.

[0365] "Subject matter" refers to the item or data that is the subject of analysis or manipulation.

[0366] "Extraction" is the act of taking useful information or elements from the original data.

[0367] "Analysis" is the process of examining data and phenomena in detail to clarify their structure and meaning.

[0368] "Information format" refers to the method of presenting data in a way that is easy to process visually or mechanically.

[0369] "Structured information" refers to data that is organized in an orderly manner and follows a specific format or pattern.

[0370] "Presentation" means the act of making information public or displaying it in a visual or other form.

[0371] "Distant" refers to a location or point that is physically far away.

[0372] The system implementing this invention mainly consists of an image acquisition device, a server, and a user terminal. After receiving image data from the user, the server performs data preprocessing and character recognition using OpenCV and Tesseract. Furthermore, it analyzes the user's emotional state from the images and audio using TensorFlow and Keras. The analyzed emotional information is presented to the user in a structured format, and the most appropriate product information and suggestions are instantly provided. Through this series of processes, the user can obtain a personalized shopping experience via a dedicated smartphone application.

[0373] For example, suppose a user scans a receipt after shopping at a store. The server uses OCR technology to convert the receipt information into text data, and then uses TensorFlow to analyze the user's emotions from their facial image based on that information. If the system determines that the user is tired, it will then present promotions for relaxing products and services in the app based on that information.

[0374] The application can generate prompt messages using a generative AI model. An example of a prompt message that would allow this application to function effectively is: "The user is in a state of high stress. Generate a prompt suggesting a product or service that would be effective in promoting relaxation." This example prompt message provides the foundation for the system to take actions in response to the user's emotional state.

[0375] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0376] Step 1:

[0377] The user launches the smartphone app and takes a picture of the receipt after making a purchase. The input is the image data of the photographed receipt. This image data is temporarily stored within the app.

[0378] Step 2:

[0379] The device sends image data acquired through the app to the server. The data sent is the raw data of the receipt image. The server receives the image data.

[0380] Step 3:

[0381] The server uses OpenCV to preprocess image data. The input is receipt image data, and the output is a clear image with noise reduction. This processing improves the image quality.

[0382] Step 4:

[0383] The server uses Tesseract with OCR technology to recognize characters from pre-processed images and extract text data. The input is pre-processed image data, and the output is extracted text data. The character information is digitized.

[0384] Step 5:

[0385] The server receives the text data obtained from character recognition and converts it into an informational format. This output is in a structured data format, such as a spreadsheet. The data is organized into a format that can be viewed at a glance.

[0386] Step 6:

[0387] The server uses TensorFlow and Keras to analyze emotions from the user's facial image or voice data. The input is the user's facial image or voice data, and the output is the emotion analysis result. The user's emotional state is identified.

[0388] Step 7:

[0389] The server uses a generative AI model based on analyzed sentiment data to provide the user with the most relevant information. This is done according to guidelines in the generated prompt text. The user is then provided with the most appropriate product information and promotions.

[0390] Step 8:

[0391] The device receives information sent from the server and displays it on the smartphone's screen. The output consists of information and campaign information customized for the user. Users can enjoy the most suitable offers based on their purchase history and reward information.

[0392] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0393] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0394] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0395] [Third Embodiment]

[0396] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0397] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0398] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0399] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0400] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0401] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0402] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0403] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0404] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0405] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0406] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0407] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0408] This invention provides a system that allows users to convert information from paper documents into digital data. First, the user uses a smartphone or tablet as an image acquisition means to take pictures of the paper documents, such as receipts or invoices. The terminal collects this image data and transmits it to a server.

[0409] The server uses image processing equipment to preprocess the received image. Specifically, it removes noise from the image and adjusts the contrast to improve reading accuracy. Then, it automatically crops the important parts of the receipt and performs OCR processing using character recognition equipment. This extracts text data from the image.

[0410] The extracted text data is organized into categories such as item name, quantity, and price using data structuring methods on the server. This data is converted into a spreadsheet format and stored in a way that makes it usable with other business tools and databases. This spreadsheet data is then transmitted to the terminal using communication methods and displayed to the user.

[0411] As a concrete example, if this system is implemented in the retail industry for expense management, users can take a picture of the paper receipt they receive at the store, instantly digitizing the purchase information and importing it into the accounting system. This process is expected to improve the accuracy of data entry and reduce manual work, thereby increasing operational efficiency.

[0412] In this system, users, terminals, and servers work together to ensure that information is accurately digitized and managed, thereby facilitating a smooth transition from paper documents to digital data.

[0413] The following describes the processing flow.

[0414] Step 1:

[0415] Users use their smartphones to launch a dedicated app and take photos of receipts and invoices with their camera. The captured images are reviewed within the app, and if there are no problems, the images are temporarily saved on the device.

[0416] Step 2:

[0417] The terminal performs basic preprocessing on the stored image data. This preprocessing includes image compression, format conversion, and encryption for communication. The processed images are then sent to the server.

[0418] Step 3:

[0419] The server analyzes the image data received from the terminal. Image processing tools are used to remove noise from the image and adjust the resolution to improve reading accuracy. Distortion correction and binning are also performed during this process.

[0420] Step 4:

[0421] The server applies OCR (Optical Character Recognition) technology to the image-processed data. Here, the character recognition means extracts the strings written on the receipt as text data and classifies them according to the purchased items.

[0422] Step 5:

[0423] The server structures the acquired text data using data structuring methods. Product names and prices are organized in a tabular format and converted to a format suitable for spreadsheets. Automatic calculations are also performed for items requiring calculations (e.g., total amount).

[0424] Step 6:

[0425] The server uses communication methods to send structured data to the terminal. The data is exported in a format that suits the user's needs, such as JSON or CSV.

[0426] Step 7:

[0427] The terminal displays the spreadsheet data received from the server and allows the user to review it. The user can then edit, save, or integrate this data with other business systems as needed.

[0428] (Example 1)

[0429] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0430] There is a need to quickly and accurately convert information recorded on paper into digital data for management and utilization. Traditional manual data entry is time-consuming and prone to errors, making it a challenge to improve efficiency and accuracy.

[0431] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0432] In this invention, the server includes data acquisition means for receiving visual data from an image acquisition device, visual processing means for pre-processing the visual data and extracting important parts of the information medium, and text recognition means for automatically analyzing text information from the extracted visuals of the information medium and converting it into electronic data format. This enables rapid and accurate conversion from paper media to digital data.

[0433] An "image acquisition device" is a hardware device used to acquire information from paper documents as digital data.

[0434] "Visual data" refers to information acquired by an image acquisition device, represented in digital format.

[0435] "Data acquisition means" refers to a function for receiving visual data transmitted from an image acquisition device.

[0436] A "visual processing means" is a function that preprocesses visual data to extract only the necessary information.

[0437] An "information medium" refers to an object or information on which text or images are recorded.

[0438] "Important parts" refer to specific sections of information recorded on an information medium that require digital conversion.

[0439] A "text recognition means" is a function that analyzes text information from visual data and converts it into an electronic data format.

[0440] An "electronic data format" is a format used to digitize information.

[0441] A "data organization method" is a function for organizing electronic data into a specific format and providing it as usable data.

[0442] A "data transmission means" is a communication function for transferring organized data to other devices or systems.

[0443] The system for carrying out the present invention is configured by combining multiple hardware and software components to efficiently collect, process, and digitize visual data.

[0444] Users acquire images of paper documents using smartphones, tablets, or other image acquisition devices. This allows users to transmit the information from the paper documents as digital data to a server. Since this transmission utilizes the device's communication capabilities, Wi-Fi and mobile networks can be used.

[0445] The server first preprocesses the received visual data using image processing software. Software such as OpenCV or ImageMagick is used for this preprocessing. Preprocessing removes noise and adjusts contrast to enable more accurate recognition of text information. Next, the server extracts text data from the image using OCR (Optical Character Recognition) software such as Tesseract.

[0446] Furthermore, the extracted text data is structured using a database management system. Specifically, it is organized into items such as item name, quantity, and price, and then converted and saved in spreadsheet format. Database software such as MySQL or PostgreSQL is used for this purpose.

[0447] The server ultimately sends the structured spreadsheet data to the device. On the device, this data is displayed in an application such as Excel or Google Sheets, allowing the user to view and manipulate the data.

[0448] As a concrete example, imagine a user taking a picture of a paper receipt at a retail store and digitizing that information for expense management. By using this system, users can efficiently improve the accuracy of data entry and streamline their business processes.

[0449] A possible prompt for the generative AI model would be, "Please tell me how to convert paper receipts into digital data and manage them efficiently."

[0450] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0451] Step 1:

[0452] The user takes a photograph of a paper document using a smartphone or tablet, which is an image acquisition device. This image data is then stored on the device. The input is paper, and the output is digital image data. Specifically, the user launches a camera app and takes a picture of a receipt or invoice.

[0453] Step 2:

[0454] The device sends stored image data to the server. The input is digital image data stored on the device, and the output is the data sent to the server. Here, the device uses Wi-Fi or mobile data communication to send data to the server over the network.

[0455] Step 3:

[0456] The server performs preprocessing on the received image data. Specifically, it performs noise reduction and contrast adjustment. The input is the transmitted image data, and the output is the image data with noise reduction and contrast adjustment. Image processing software such as ImageMagick and OpenCV is used to perform processing to improve visibility.

[0457] Step 4:

[0458] The server extracts text information from pre-processed image data. Here, character recognition is performed using OCR technology. The input is pre-processed image data, and the output is text data. This involves the specific operation of converting the text portion of the image into text format using OCR software such as Tesseract.

[0459] Step 5:

[0460] The server structures the extracted text data. It organizes the text data into categories such as item name, quantity, and price, and converts it into a spreadsheet format. The input is unorganized text data, and the output is structured spreadsheet data. A database management system such as MySQL or PostgreSQL is used to appropriately classify and store the data.

[0461] Step 6:

[0462] The server sends structured data to the terminal, and the user views and reviews this data. The input is structured spreadsheet data, and the output is data displayed on the terminal. The data can be viewed and used by the user through applications such as Excel or Google Sheets.

[0463] (Application Example 1)

[0464] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0465] Current online shopping return procedures are cumbersome and time-consuming for buyers. Furthermore, manually entering receipt information increases the likelihood of human error. Therefore, it is necessary to simplify the return process and automate it using digital information.

[0466] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0467] In this invention, the server includes an image acquisition means for receiving image data from an image acquisition device, an image processing means for automatically extracting important information, a character recognition means for converting character information into a digital data format, a data structuring means for linking purchase information and outputting data in tab format, and a communication means for automating the return process. This allows users to perform the return procedure smoothly and significantly reduce the time and effort required.

[0468] "Image acquisition means" refers to the technical elements for receiving image data from a device.

[0469] "Image processing means" refers to technical elements that perform preprocessing on image data and automatically extract important information.

[0470] "Character recognition means" refers to the technical elements used to analyze character information from extracted information and convert it into digital data format.

[0471] "Data structuring means" refers to technical elements for linking text information with purchase information and outputting it in a structured tab format.

[0472] "Communication means" refers to the technical elements used to transmit tab-formatted data to a remote computing device and automate the return processing.

[0473] The system designed to realize this application seamlessly and efficiently supports users in the return process using a smartphone app.

[0474] The user takes a picture of the receipt for the item they want to return using their smartphone camera. The captured image is optimized by an application on the device and sent to the server. The server uses an image processing library (e.g., OpenCV) to remove noise from the received image data, crops the necessary areas, and extracts important information. Next, an OCR engine (e.g., Tesseract OCR) is used to analyze the text information from the image and convert it into digital data.

[0475] The converted digital data is linked to purchase information and stored in a database. A data processing library (e.g., Pandas) is used to structure the data. It is then organized as tabbed data and transmitted to a remote computing device via a REST API. Finally, the user can confirm that the return process proceeds automatically within the application.

[0476] As a concrete example of this system, consider a case where a user purchases clothing online but the size is incorrect. In this case, the user takes a picture of the receipt for the clothing and selects to proceed with the return process within the app. This entire process significantly simplifies the return process and improves the user experience. As an example of a prompt message to the generating AI model, instructions such as "I would like to return this item. I have taken a picture of the receipt, so please proceed with the return process" can be entered into the application.

[0477] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0478] Step 1:

[0479] The terminal takes a picture of the receipt for the item the user wishes to return using the smartphone's camera. The input is the image of the photographed receipt, and the output is the image data stored on the terminal. The user's photography operation is seamlessly handled, and a process to verify the image resolution is performed.

[0480] Step 2:

[0481] The terminal optimizes the captured image data and sends it to the server. The input is the image data obtained in step 1, and the output is the optimized image data sent to the server. Specifically, the terminal compresses the image, adjusts the resolution, and sends the data via a communication protocol.

[0482] Step 3:

[0483] The server performs image processing on the received image data. The input is the image data received by the server, and the output is the denoised and trimmed image data. The server uses OpenCV to perform denoising and trim important parts.

[0484] Step 4:

[0485] The server analyzes the characters from the cropped image and converts them into digital data. The input is the cropped image from step 3, and the output is digital data in the form of text. The server performs OCR processing using Tesseract OCR to obtain the character information.

[0486] Step 5:

[0487] The server structures the retrieved text information and links it to purchase information. The input is the text data obtained in step 4, and the output is structured tab-formatted data. The server uses the Pandas library to organize the data and stores the purchased item names and price information in the database.

[0488] Step 6:

[0489] The server sends structured data to a remote computing device using a communication method. The input is tab-formatted data generated in step 5, and the output is data processed by the remote system. The server sends data via a REST API and is configured to automatically handle return processing.

[0490] Step 7:

[0491] The user confirms the completion of the return process on their smartphone application. The input is the processing result displayed on the device, and the output is the user's confirmation that the return process is complete. The user checks the in-app notification and recognizes that the process has been completed smoothly.

[0492] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0493] This invention is characterized by providing a system that not only allows users to convert various types of information into digital data, but also analyzes users' emotions in real time. First, the user takes a picture of a receipt or document with a smart device equipped with image acquisition means, and temporarily stores the image data on the terminal. The terminal sends this image to a server, and the server analyzes the image using image processing means.

[0494] On the server, OCR technology is applied to the received image data to perform character recognition and extract information written on paper as text data. Subsequently, the purchased item information is organized into a spreadsheet format using a data structuring method. At this time, the emotion engine is activated and analyzes the user's emotional state using the image and audio data obtained from the user.

[0495] The analysis results from the emotion engine are reflected in the customization of data output. For example, if a user is experiencing stress, the system can simplify the interface and display designs and suggestions that reduce stress. It also has a mechanism to adjust the priority and display format of data presentation based on emotions, quickly providing the user with the information they need most.

[0496] For example, if a user scans documents under tight time pressure before a meeting, the emotion engine detects the time-pressured and stressed state, and the device provides highlights of important information and priority notifications. This allows the user to quickly and accurately obtain the necessary information.

[0497] By adding an emotion engine, this system transcends being merely a data conversion tool, providing support functions that enhance the user experience. This enables users to manage information efficiently and with less stress.

[0498] The following describes the processing flow.

[0499] Step 1:

[0500] Users use a device such as a smartphone or tablet to launch the application and take a picture of a paper receipt or document. After taking the picture, the app temporarily saves the image to the device. At this point, users can also use options such as taking a selfie or voice input for emotion analysis.

[0501] Step 2:

[0502] The device sends captured image data and additional data from the user to the server. Before transmission, the data is compressed and encrypted to ensure secure communication. Images and audio are also transmitted as emotional data.

[0503] Step 3:

[0504] The server performs image processing on the image data received from the terminal. Specifically, it removes noise and crops the data, extracting only the necessary parts. In parallel, the emotion engine begins analyzing the user's emotions. This analysis is based on the received selfie images and audio information.

[0505] Step 4:

[0506] The server uses OCR technology to extract text data from images. The recognized text information is classified into categories such as purchase information and numerical values. Taking into account the results of the emotion engine's analysis, the data is organized into a spreadsheet format using data structuring methods.

[0507] Step 5:

[0508] The emotion engine instructs the user interface (UI) to be customized based on the analysis results. When the user is relaxed, it provides information using the normal UI, while when the user is experiencing stress, it prioritizes displaying important information. It also dynamically adjusts the priority of data output according to the user's emotions.

[0509] Step 6:

[0510] The server sends organized spreadsheet data and customized UI information to the device. The data is output in the format specified by the user and can be downloaded as CSV or JSON as needed.

[0511] Step 7:

[0512] The device displays data received from the server and UI information to the user. The user can review this data and, if necessary, edit, save, and share it. Furthermore, the displayed information is adjusted based on sentiment analysis, improving the user's information access experience.

[0513] (Example 2)

[0514] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0515] The problem that this invention aims to solve is to reduce user stress and enable efficient information management by providing adaptive data presentation that corresponds to the user's emotional state during the conversion process to digital data.

[0516] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0517] In this invention, the server includes acquisition means for receiving image information, analysis means for preprocessing, recognition means for converting character information into a data format, adjustment means for customizing data based on emotional state, and transmission means for transmitting data. This enables the display and management of information corresponding to the user's emotional state.

[0518] "Acquisition means" refers to a function or device for receiving image information from an information processing device.

[0519] "Analysis means" refers to a function or device for preprocessing received image information and extracting specific target information.

[0520] "Recognition means" refers to a function or device for analyzing textual information from extracted images and converting it into a data format.

[0521] A "processing method" is a function or device for organizing converted character information as structured data and outputting it in tabular format.

[0522] "Adjustment means" refers to a function or device that adaptively customizes data output based on the results of an analysis of the user's emotional state.

[0523] "Transmission means" refers to a function or device for securely transmitting adjusted data to a remote information processing device.

[0524] This system enables users to effectively digitize information and manage it using sentiment analysis. Users first use a smart device with image acquisition capabilities to photograph documents or receipts. The image data is temporarily stored on the smart device and then transmitted to the server using a secure communication protocol (e.g., HTTPS).

[0525] The server preprocesses images using an image processing library (e.g., OpenCV) to remove noise and correct tilt. Next, it uses OCR technology (e.g., Tesseract OCR) to identify text information and extract text data from the images. This text data is then organized into a spreadsheet format using a data structuring library (e.g., Python's Pandas).

[0526] Furthermore, the server uses an emotion analysis engine to analyze the emotional state from image and audio data provided by the user. For example, by utilizing natural language processing tools (e.g., Google Cloud Natural Language API), the user's emotional state can be understood in real time. Based on the results of the emotion analysis, the user interface is adjusted, and information that matches the user's needs is prioritized. For example, important information can be concisely summarized and highlighted for users who are feeling stressed.

[0527] For example, if a user scans a document while pressed for time before a meeting, the system detects this state of urgency and highlights important information for immediate access. In this way, the user can quickly and accurately obtain the necessary information.

[0528] For example, entering a prompt such as, "I scanned the documents before the meeting. I'm busy, so please prioritize showing me the most important information," will allow the system to provide more relevant information.

[0529] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0530] Step 1:

[0531] The user uses a smart device to photograph receipts or documents. The input for this procedure is the physical paper documents themselves. When the user photographs them, they are output as image data and temporarily stored on the smart device.

[0532] Step 2:

[0533] The terminal sends image data stored on the smart device to the server. The input for this procedure is the image data stored on the terminal. The terminal sends the image data to the server, and the image data is transferred to the server as output using a secure communication protocol (e.g., HTTPS).

[0534] Step 3:

[0535] The server analyzes the image data it receives. The input to this process is the image data that arrives at the server. The server uses an image processing library (e.g., OpenCV) to perform preprocessing such as noise reduction and tilt correction, and outputs the preprocessed image data.

[0536] Step 4:

[0537] The server extracts text information from image data using OCR technology. The input for this procedure is pre-processed image data. The server applies OCR (e.g., Tesseract OCR) to extract the characters in the image as text data and outputs that text data.

[0538] Step 5:

[0539] The server organizes text data into a spreadsheet format. The input for this process is the text data extracted by the server in a previous step. The server uses a data organization library (e.g., Python's Pandas) to structure the text data and output it as a spreadsheet.

[0540] Step 6:

[0541] The server uses an emotion analysis engine to analyze the user's emotions. The user's image and audio data are used as input for the emotion analysis. The server utilizes natural language processing tools (e.g., Google Cloud Natural Language API) to analyze the user's emotional state and outputs the results.

[0542] Step 7:

[0543] The server customizes the data output based on the analysis results. The input for this procedure is the analysis results from the sentiment analysis engine. Based on this, the server adjusts the interface design and how information is presented, and outputs customized data adapted to the user.

[0544] (Application Example 2)

[0545] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0546] Many modern consumers seek ways to manage information about their purchased products and optimize their purchasing experience according to their individual emotional states. However, conventional information management systems struggle to consider users' emotional states and fail to adequately improve the consumer experience. In this context, there is a need to develop systems that enable flexible and appropriate information presentation based on users' emotions.

[0547] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0548] In this invention, the server includes means for receiving image data from an image acquisition device, processing means for pre-processing the image data and extracting specific parts of an object, and recognition means for analyzing textual information from the extracted image of the object and converting it into an information format. This makes it possible to easily organize information about products purchased by consumers, and further analyze the user's emotional state based on that information, enabling the presentation of information tailored to their emotions.

[0549] A "device" is a piece of equipment used to acquire or process information.

[0550] "Means" refer to methods or components used to achieve a specific function or purpose.

[0551] "Image data" refers to visual information represented in electronic format.

[0552] "Preprocessing" refers to preparatory work performed before data analysis, and is carried out to improve the quality of the data.

[0553] "Subject matter" refers to the item or data that is the subject of analysis or manipulation.

[0554] "Extraction" is the act of taking useful information or elements from the original data.

[0555] "Analysis" is the process of examining data and phenomena in detail to clarify their structure and meaning.

[0556] "Information format" refers to the method of presenting data in a way that is easy to process visually or mechanically.

[0557] "Structured information" refers to data that is organized in an orderly manner and follows a specific format or pattern.

[0558] "Presentation" means the act of making information public or displaying it in a visual or other form.

[0559] "Distant" refers to a location or point that is physically far away.

[0560] The system implementing this invention mainly consists of an image acquisition device, a server, and a user terminal. After receiving image data from the user, the server performs data preprocessing and character recognition using OpenCV and Tesseract. Furthermore, it analyzes the user's emotional state from the images and audio using TensorFlow and Keras. The analyzed emotional information is presented to the user in a structured format, and the most appropriate product information and suggestions are instantly provided. Through this series of processes, the user can obtain a personalized shopping experience via a dedicated smartphone application.

[0561] For example, suppose a user scans a receipt after shopping at a store. The server uses OCR technology to convert the receipt information into text data, and then uses TensorFlow to analyze the user's emotions from their facial image based on that information. If the system determines that the user is tired, it will then present promotions for relaxing products and services in the app based on that information.

[0562] The application can generate prompt messages using a generative AI model. An example of a prompt message that would allow this application to function effectively is: "The user is in a state of high stress. Generate a prompt suggesting a product or service that would be effective in promoting relaxation." This example prompt message provides the foundation for the system to take actions in response to the user's emotional state.

[0563] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0564] Step 1:

[0565] The user launches the smartphone app and takes a picture of the receipt after making a purchase. The input is the image data of the photographed receipt. This image data is temporarily stored within the app.

[0566] Step 2:

[0567] The device sends image data acquired through the app to the server. The data sent is the raw data of the receipt image. The server receives the image data.

[0568] Step 3:

[0569] The server uses OpenCV to preprocess image data. The input is receipt image data, and the output is a clear image with noise reduction. This processing improves the image quality.

[0570] Step 4:

[0571] The server uses Tesseract with OCR technology to recognize characters from pre-processed images and extract text data. The input is pre-processed image data, and the output is extracted text data. The character information is digitized.

[0572] Step 5:

[0573] The server receives the text data obtained from character recognition and converts it into an informational format. This output is in a structured data format, such as a spreadsheet. The data is organized into a format that can be viewed at a glance.

[0574] Step 6:

[0575] The server uses TensorFlow and Keras to analyze emotions from the user's facial image or voice data. The input is the user's facial image or voice data, and the output is the emotion analysis result. The user's emotional state is identified.

[0576] Step 7:

[0577] The server uses a generative AI model based on analyzed sentiment data to provide the user with the most relevant information. This is done according to guidelines in the generated prompt text. The user is then provided with the most appropriate product information and promotions.

[0578] Step 8:

[0579] The device receives information sent from the server and displays it on the smartphone's screen. The output consists of information and campaign information customized for the user. Users can enjoy the most suitable offers based on their purchase history and reward information.

[0580] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0581] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0582] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0583] [Fourth Embodiment]

[0584] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0585] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0586] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0587] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0588] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0589] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0590] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0591] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0592] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0593] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0594] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0595] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0596] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0597] This invention provides a system that allows users to convert information from paper documents into digital data. First, the user uses a smartphone or tablet as an image acquisition means to take pictures of the paper documents, such as receipts or invoices. The terminal collects this image data and transmits it to a server.

[0598] The server uses image processing equipment to preprocess the received image. Specifically, it removes noise from the image and adjusts the contrast to improve reading accuracy. Then, it automatically crops the important parts of the receipt and performs OCR processing using character recognition equipment. This extracts text data from the image.

[0599] The extracted text data is organized into categories such as item name, quantity, and price using data structuring methods on the server. This data is converted into a spreadsheet format and stored in a way that makes it usable with other business tools and databases. This spreadsheet data is then transmitted to the terminal using communication methods and displayed to the user.

[0600] As a concrete example, if this system is implemented in the retail industry for expense management, users can take a picture of the paper receipt they receive at the store, instantly digitizing the purchase information and importing it into the accounting system. This process is expected to improve the accuracy of data entry and reduce manual work, thereby increasing operational efficiency.

[0601] In this system, users, terminals, and servers work together to ensure that information is accurately digitized and managed, thereby facilitating a smooth transition from paper documents to digital data.

[0602] The following describes the processing flow.

[0603] Step 1:

[0604] Users use their smartphones to launch a dedicated app and take photos of receipts and invoices with their camera. The captured images are reviewed within the app, and if there are no problems, the images are temporarily saved on the device.

[0605] Step 2:

[0606] The terminal performs basic preprocessing on the stored image data. This preprocessing includes image compression, format conversion, and encryption for communication. The processed images are then sent to the server.

[0607] Step 3:

[0608] The server analyzes the image data received from the terminal. Image processing tools are used to remove noise from the image and adjust the resolution to improve reading accuracy. Distortion correction and binning are also performed during this process.

[0609] Step 4:

[0610] The server applies OCR (Optical Character Recognition) technology to the image-processed data. Here, the character recognition means extracts the strings written on the receipt as text data and classifies them according to the purchased items.

[0611] Step 5:

[0612] The server structures the acquired text data using data structuring methods. Product names and prices are organized in a tabular format and converted to a format suitable for spreadsheets. Automatic calculations are also performed for items requiring calculations (e.g., total amount).

[0613] Step 6:

[0614] The server uses communication methods to send structured data to the terminal. The data is exported in a format that suits the user's needs, such as JSON or CSV.

[0615] Step 7:

[0616] The terminal displays the spreadsheet data received from the server and allows the user to review it. The user can then edit, save, or integrate this data with other business systems as needed.

[0617] (Example 1)

[0618] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0619] There is a need to quickly and accurately convert information recorded on paper into digital data for management and utilization. Traditional manual data entry is time-consuming and prone to errors, making it a challenge to improve efficiency and accuracy.

[0620] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0621] In this invention, the server includes data acquisition means for receiving visual data from an image acquisition device, visual processing means for pre-processing the visual data and extracting important parts of the information medium, and text recognition means for automatically analyzing text information from the extracted visuals of the information medium and converting it into electronic data format. This enables rapid and accurate conversion from paper media to digital data.

[0622] An "image acquisition device" is a hardware device used to acquire information from paper documents as digital data.

[0623] "Visual data" refers to information acquired by an image acquisition device, represented in digital format.

[0624] "Data acquisition means" refers to a function for receiving visual data transmitted from an image acquisition device.

[0625] A "visual processing means" is a function that preprocesses visual data to extract only the necessary information.

[0626] An "information medium" refers to an object or information on which text or images are recorded.

[0627] "Important parts" refer to specific sections of information recorded on an information medium that require digital conversion.

[0628] A "text recognition means" is a function that analyzes text information from visual data and converts it into an electronic data format.

[0629] An "electronic data format" is a format used to digitize information.

[0630] A "data organization method" is a function for organizing electronic data into a specific format and providing it as usable data.

[0631] A "data transmission means" is a communication function for transferring organized data to other devices or systems.

[0632] The system for carrying out the present invention is configured by combining multiple hardware and software components to efficiently collect, process, and digitize visual data.

[0633] Users acquire images of paper documents using smartphones, tablets, or other image acquisition devices. This allows users to transmit the information from the paper documents as digital data to a server. Since this transmission utilizes the device's communication capabilities, Wi-Fi and mobile networks can be used.

[0634] The server first preprocesses the received visual data using image processing software. Software such as OpenCV or ImageMagick is used for this preprocessing. Preprocessing removes noise and adjusts contrast to enable more accurate recognition of text information. Next, the server extracts text data from the image using OCR (Optical Character Recognition) software such as Tesseract.

[0635] Furthermore, the extracted text data is structured using a database management system. Specifically, it is organized into items such as item name, quantity, and price, and then converted and saved in spreadsheet format. Database software such as MySQL or PostgreSQL is used for this purpose.

[0636] The server ultimately sends the structured spreadsheet data to the device. On the device, this data is displayed in an application such as Excel or Google Sheets, allowing the user to view and manipulate the data.

[0637] As a concrete example, imagine a user taking a picture of a paper receipt at a retail store and digitizing that information for expense management. By using this system, users can efficiently improve the accuracy of data entry and streamline their business processes.

[0638] A possible prompt for the generative AI model would be, "Please tell me how to convert paper receipts into digital data and manage them efficiently."

[0639] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0640] Step 1:

[0641] The user takes a photograph of a paper document using a smartphone or tablet, which is an image acquisition device. This image data is then stored on the device. The input is paper, and the output is digital image data. Specifically, the user launches a camera app and takes a picture of a receipt or invoice.

[0642] Step 2:

[0643] The device sends stored image data to the server. The input is digital image data stored on the device, and the output is the data sent to the server. Here, the device uses Wi-Fi or mobile data communication to send data to the server over the network.

[0644] Step 3:

[0645] The server performs preprocessing on the received image data. Specifically, it performs noise reduction and contrast adjustment. The input is the transmitted image data, and the output is the image data with noise reduction and contrast adjustment. Image processing software such as ImageMagick and OpenCV is used to perform processing to improve visibility.

[0646] Step 4:

[0647] The server extracts text information from pre-processed image data. Here, character recognition is performed using OCR technology. The input is pre-processed image data, and the output is text data. This involves the specific operation of converting the text portion of the image into text format using OCR software such as Tesseract.

[0648] Step 5:

[0649] The server structures the extracted text data. It organizes the text data into categories such as item name, quantity, and price, and converts it into a spreadsheet format. The input is unorganized text data, and the output is structured spreadsheet data. A database management system such as MySQL or PostgreSQL is used to appropriately classify and store the data.

[0650] Step 6:

[0651] The server sends structured data to the terminal, and the user views and reviews this data. The input is structured spreadsheet data, and the output is data displayed on the terminal. The data can be viewed and used by the user through applications such as Excel or Google Sheets.

[0652] (Application Example 1)

[0653] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0654] Current online shopping return procedures are cumbersome and time-consuming for buyers. Furthermore, manually entering receipt information increases the likelihood of human error. Therefore, it is necessary to simplify the return process and automate it using digital information.

[0655] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0656] In this invention, the server includes an image acquisition means for receiving image data from an image acquisition device, an image processing means for automatically extracting important information, a character recognition means for converting character information into a digital data format, a data structuring means for linking purchase information and outputting data in tab format, and a communication means for automating the return process. This allows users to perform the return procedure smoothly and significantly reduce the time and effort required.

[0657] "Image acquisition means" refers to the technical elements for receiving image data from a device.

[0658] "Image processing means" refers to technical elements that perform preprocessing on image data and automatically extract important information.

[0659] "Character recognition means" refers to the technical elements used to analyze character information from extracted information and convert it into digital data format.

[0660] "Data structuring means" refers to technical elements for linking text information with purchase information and outputting it in a structured tab format.

[0661] "Communication means" refers to the technical elements used to transmit tab-formatted data to a remote computing device and automate the return processing.

[0662] The system designed to realize this application seamlessly and efficiently supports users in the return process using a smartphone app.

[0663] The user takes a picture of the receipt for the item they want to return using their smartphone camera. The captured image is optimized by an application on the device and sent to the server. The server uses an image processing library (e.g., OpenCV) to remove noise from the received image data, crops the necessary areas, and extracts important information. Next, an OCR engine (e.g., Tesseract OCR) is used to analyze the text information from the image and convert it into digital data.

[0664] The converted digital data is linked to purchase information and stored in a database. A data processing library (e.g., Pandas) is used to structure the data. It is then organized as tabbed data and transmitted to a remote computing device via a REST API. Finally, the user can confirm that the return process proceeds automatically within the application.

[0665] As a concrete example of this system, consider a case where a user purchases clothing online but the size is incorrect. In this case, the user takes a picture of the receipt for the clothing and selects to proceed with the return process within the app. This entire process significantly simplifies the return process and improves the user experience. As an example of a prompt message to the generating AI model, instructions such as "I would like to return this item. I have taken a picture of the receipt, so please proceed with the return process" can be entered into the application.

[0666] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0667] Step 1:

[0668] The terminal takes a picture of the receipt for the item the user wishes to return using the smartphone's camera. The input is the image of the photographed receipt, and the output is the image data stored on the terminal. The user's photography operation is seamlessly handled, and a process to verify the image resolution is performed.

[0669] Step 2:

[0670] The terminal optimizes the captured image data and sends it to the server. The input is the image data obtained in step 1, and the output is the optimized image data sent to the server. Specifically, the terminal compresses the image, adjusts the resolution, and sends the data via a communication protocol.

[0671] Step 3:

[0672] The server performs image processing on the received image data. The input is the image data received by the server, and the output is the denoised and trimmed image data. The server uses OpenCV to perform denoising and trim important parts.

[0673] Step 4:

[0674] The server analyzes the characters from the cropped image and converts them into digital data. The input is the cropped image from step 3, and the output is digital data in the form of text. The server performs OCR processing using Tesseract OCR to obtain the character information.

[0675] Step 5:

[0676] The server structures the retrieved text information and links it to purchase information. The input is the text data obtained in step 4, and the output is structured tab-formatted data. The server uses the Pandas library to organize the data and stores the purchased item names and price information in the database.

[0677] Step 6:

[0678] The server sends structured data to a remote computing device using a communication method. The input is tab-formatted data generated in step 5, and the output is data processed by the remote system. The server sends data via a REST API and is configured to automatically handle return processing.

[0679] Step 7:

[0680] The user confirms the completion of the return process on their smartphone application. The input is the processing result displayed on the device, and the output is the user's confirmation that the return process is complete. The user checks the in-app notification and recognizes that the process has been completed smoothly.

[0681] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0682] This invention is characterized by providing a system that not only allows users to convert various types of information into digital data, but also analyzes users' emotions in real time. First, the user takes a picture of a receipt or document with a smart device equipped with image acquisition means, and temporarily stores the image data on the terminal. The terminal sends this image to a server, and the server analyzes the image using image processing means.

[0683] On the server, OCR technology is applied to the received image data to perform character recognition and extract information written on paper as text data. Subsequently, the purchased item information is organized into a spreadsheet format using a data structuring method. At this time, the emotion engine is activated and analyzes the user's emotional state using the image and audio data obtained from the user.

[0684] The analysis results from the emotion engine are reflected in the customization of data output. For example, if a user is experiencing stress, the system can simplify the interface and display designs and suggestions that reduce stress. It also has a mechanism to adjust the priority and display format of data presentation based on emotions, quickly providing the user with the information they need most.

[0685] For example, if a user scans documents under tight time pressure before a meeting, the emotion engine detects the time-pressured and stressed state, and the device provides highlights of important information and priority notifications. This allows the user to quickly and accurately obtain the necessary information.

[0686] By adding an emotion engine, this system transcends being merely a data conversion tool, providing support functions that enhance the user experience. This enables users to manage information efficiently and with less stress.

[0687] The following describes the processing flow.

[0688] Step 1:

[0689] Users use a device such as a smartphone or tablet to launch the application and take a picture of a paper receipt or document. After taking the picture, the app temporarily saves the image to the device. At this point, users can also use options such as taking a selfie or voice input for emotion analysis.

[0690] Step 2:

[0691] The device sends captured image data and additional data from the user to the server. Before transmission, the data is compressed and encrypted to ensure secure communication. Images and audio are also transmitted as emotional data.

[0692] Step 3:

[0693] The server performs image processing on the image data received from the terminal. Specifically, it removes noise and crops the data, extracting only the necessary parts. In parallel, the emotion engine begins analyzing the user's emotions. This analysis is based on the received selfie images and audio information.

[0694] Step 4:

[0695] The server uses OCR technology to extract text data from images. The recognized text information is classified into categories such as purchase information and numerical values. Taking into account the results of the emotion engine's analysis, the data is organized into a spreadsheet format using data structuring methods.

[0696] Step 5:

[0697] The emotion engine instructs the user interface (UI) to be customized based on the analysis results. When the user is relaxed, it provides information using the normal UI, while when the user is experiencing stress, it prioritizes displaying important information. It also dynamically adjusts the priority of data output according to the user's emotions.

[0698] Step 6:

[0699] The server sends organized spreadsheet data and customized UI information to the device. The data is output in the format specified by the user and can be downloaded as CSV or JSON as needed.

[0700] Step 7:

[0701] The device displays data received from the server and UI information to the user. The user can review this data and, if necessary, edit, save, and share it. Furthermore, the displayed information is adjusted based on sentiment analysis, improving the user's information access experience.

[0702] (Example 2)

[0703] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0704] The problem that this invention aims to solve is to reduce user stress and enable efficient information management by providing adaptive data presentation that corresponds to the user's emotional state during the conversion process to digital data.

[0705] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0706] In this invention, the server includes acquisition means for receiving image information, analysis means for preprocessing, recognition means for converting character information into a data format, adjustment means for customizing data based on emotional state, and transmission means for transmitting data. This enables the display and management of information corresponding to the user's emotional state.

[0707] "Acquisition means" refers to a function or device for receiving image information from an information processing device.

[0708] "Analysis means" refers to a function or device for preprocessing received image information and extracting specific target information.

[0709] "Recognition means" refers to a function or device for analyzing textual information from extracted images and converting it into a data format.

[0710] A "processing method" is a function or device for organizing converted character information as structured data and outputting it in tabular format.

[0711] "Adjustment means" refers to a function or device that adaptively customizes data output based on the results of an analysis of the user's emotional state.

[0712] "Transmission means" refers to a function or device for securely transmitting adjusted data to a remote information processing device.

[0713] This system enables users to effectively digitize information and manage it using sentiment analysis. Users first use a smart device with image acquisition capabilities to photograph documents or receipts. The image data is temporarily stored on the smart device and then transmitted to the server using a secure communication protocol (e.g., HTTPS).

[0714] The server preprocesses images using an image processing library (e.g., OpenCV) to remove noise and correct tilt. Next, it uses OCR technology (e.g., Tesseract OCR) to identify text information and extract text data from the images. This text data is then organized into a spreadsheet format using a data structuring library (e.g., Python's Pandas).

[0715] Furthermore, the server uses an emotion analysis engine to analyze the emotional state from image and audio data provided by the user. For example, by utilizing natural language processing tools (e.g., Google Cloud Natural Language API), the user's emotional state can be understood in real time. Based on the results of the emotion analysis, the user interface is adjusted, and information that matches the user's needs is prioritized. For example, important information can be concisely summarized and highlighted for users who are feeling stressed.

[0716] For example, if a user scans a document while pressed for time before a meeting, the system detects this state of urgency and highlights important information for immediate access. In this way, the user can quickly and accurately obtain the necessary information.

[0717] For example, entering a prompt such as, "I scanned the documents before the meeting. I'm busy, so please prioritize showing me the most important information," will allow the system to provide more relevant information.

[0718] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0719] Step 1:

[0720] The user uses a smart device to photograph receipts or documents. The input for this procedure is the physical paper documents themselves. When the user photographs them, they are output as image data and temporarily stored on the smart device.

[0721] Step 2:

[0722] The terminal sends image data stored on the smart device to the server. The input for this procedure is the image data stored on the terminal. The terminal sends the image data to the server, and the image data is transferred to the server as output using a secure communication protocol (e.g., HTTPS).

[0723] Step 3:

[0724] The server analyzes the image data it receives. The input to this process is the image data that arrives at the server. The server uses an image processing library (e.g., OpenCV) to perform preprocessing such as noise reduction and tilt correction, and outputs the preprocessed image data.

[0725] Step 4:

[0726] The server extracts text information from image data using OCR technology. The input for this procedure is pre-processed image data. The server applies OCR (e.g., Tesseract OCR) to extract the characters in the image as text data and outputs that text data.

[0727] Step 5:

[0728] The server organizes text data into a spreadsheet format. The input for this process is the text data extracted by the server in a previous step. The server uses a data organization library (e.g., Python's Pandas) to structure the text data and output it as a spreadsheet.

[0729] Step 6:

[0730] The server uses an emotion analysis engine to analyze the user's emotions. The user's image and audio data are used as input for the emotion analysis. The server utilizes natural language processing tools (e.g., Google Cloud Natural Language API) to analyze the user's emotional state and outputs the results.

[0731] Step 7:

[0732] The server customizes the data output based on the analysis results. The input for this procedure is the analysis results from the sentiment analysis engine. Based on this, the server adjusts the interface design and how information is presented, and outputs customized data adapted to the user.

[0733] (Application Example 2)

[0734] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0735] Many modern consumers seek ways to manage information about their purchased products and optimize their purchasing experience according to their individual emotional states. However, conventional information management systems struggle to consider users' emotional states and fail to adequately improve the consumer experience. In this context, there is a need to develop systems that enable flexible and appropriate information presentation based on users' emotions.

[0736] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0737] In this invention, the server includes means for receiving image data from an image acquisition device, processing means for pre-processing the image data and extracting specific parts of an object, and recognition means for analyzing textual information from the extracted image of the object and converting it into an information format. This makes it possible to easily organize information about products purchased by consumers, and further analyze the user's emotional state based on that information, enabling the presentation of information tailored to their emotions.

[0738] A "device" is a piece of equipment used to acquire or process information.

[0739] "Means" refer to methods or components used to achieve a specific function or purpose.

[0740] "Image data" refers to visual information represented in electronic format.

[0741] "Preprocessing" refers to preparatory work performed before data analysis, and is carried out to improve the quality of the data.

[0742] "Subject matter" refers to the item or data that is the subject of analysis or manipulation.

[0743] "Extraction" is the act of taking useful information or elements from the original data.

[0744] "Analysis" is the process of examining data and phenomena in detail to clarify their structure and meaning.

[0745] "Information format" refers to the method of presenting data in a way that is easy to process visually or mechanically.

[0746] "Structured information" refers to data that is organized in an orderly manner and follows a specific format or pattern.

[0747] "Presentation" means the act of making information public or displaying it in a visual or other form.

[0748] "Distant" refers to a location or point that is physically far away.

[0749] The system implementing this invention mainly consists of an image acquisition device, a server, and a user terminal. After receiving image data from the user, the server performs data preprocessing and character recognition using OpenCV and Tesseract. Furthermore, it analyzes the user's emotional state from the images and audio using TensorFlow and Keras. The analyzed emotional information is presented to the user in a structured format, and the most appropriate product information and suggestions are instantly provided. Through this series of processes, the user can obtain a personalized shopping experience via a dedicated smartphone application.

[0750] For example, suppose a user scans a receipt after shopping at a store. The server uses OCR technology to convert the receipt information into text data, and then uses TensorFlow to analyze the user's emotions from their facial image based on that information. If the system determines that the user is tired, it will then present promotions for relaxing products and services in the app based on that information.

[0751] The application can generate prompt messages using a generative AI model. An example of a prompt message that would allow this application to function effectively is: "The user is in a state of high stress. Generate a prompt suggesting a product or service that would be effective in promoting relaxation." This example prompt message provides the foundation for the system to take actions in response to the user's emotional state.

[0752] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0753] Step 1:

[0754] The user launches the smartphone app and takes a picture of the receipt after making a purchase. The input is the image data of the photographed receipt. This image data is temporarily stored within the app.

[0755] Step 2:

[0756] The device sends image data acquired through the app to the server. The data sent is the raw data of the receipt image. The server receives the image data.

[0757] Step 3:

[0758] The server uses OpenCV to preprocess image data. The input is receipt image data, and the output is a clear image with noise reduction. This processing improves the image quality.

[0759] Step 4:

[0760] The server uses Tesseract with OCR technology to recognize characters from pre-processed images and extract text data. The input is pre-processed image data, and the output is extracted text data. The character information is digitized.

[0761] Step 5:

[0762] The server receives the text data obtained from character recognition and converts it into an informational format. This output is in a structured data format, such as a spreadsheet. The data is organized into a format that can be viewed at a glance.

[0763] Step 6:

[0764] The server uses TensorFlow and Keras to analyze emotions from the user's facial image or voice data. The input is the user's facial image or voice data, and the output is the emotion analysis result. The user's emotional state is identified.

[0765] Step 7:

[0766] The server uses a generative AI model based on analyzed sentiment data to provide the user with the most relevant information. This is done according to guidelines in the generated prompt text. The user is then provided with the most appropriate product information and promotions.

[0767] Step 8:

[0768] The device receives information sent from the server and displays it on the smartphone's screen. The output consists of information and campaign information customized for the user. Users can enjoy the most suitable offers based on their purchase history and reward information.

[0769] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0770] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0771] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0772] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0773] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0774] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0775] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0776] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0777] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0778] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0779] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0780] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0781] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0782] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0783] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0784] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0785] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0786] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0787] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0788] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0789] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0790] The following is further disclosed regarding the embodiments described above.

[0791] (Claim 1)

[0792] An image acquisition means that receives image data from an image acquisition device,

[0793] Image processing means that preprocesses the aforementioned image data and extracts a specific part of the object,

[0794] A character recognition means that analyzes character information from the image of the extracted object and converts it into a data format,

[0795] A data structuring means that organizes the character information converted to the aforementioned data format as structured data and outputs it in tabular format,

[0796] A system including a communication means for transmitting the aforementioned tabular data to a remote device.

[0797] (Claim 2)

[0798] The system according to claim 1, characterized in that the communication means converts the tabular data into a plurality of data formats and outputs them selectively.

[0799] (Claim 3)

[0800] The system according to claim 1, characterized in that the image acquisition means acquires images by remote operation from a user device.

[0801] "Example 1"

[0802] (Claim 1)

[0803] A data acquisition means that receives visual data from an image acquisition device,

[0804] A visual processing means that preprocesses the aforementioned visual data and extracts the important parts of the information medium,

[0805] A text recognition means that automatically analyzes text information from the visuals of the extracted information medium and converts it into an electronic data format,

[0806] A data organization means that organizes the text information converted into the aforementioned electronic data format as structured data and displays the organized data in a tabular format,

[0807] A system including data transfer means for transferring the aforementioned tabular data to another computer device.

[0808] (Claim 2)

[0809] The system according to claim 1, characterized in that the data transmission means converts the tabular data into various data formats and outputs them selectively.

[0810] (Claim 3)

[0811] The system according to claim 1, characterized in that the data collection means acquires visuals by remote operation from a user device.

[0812] "Application Example 1"

[0813] (Claim 1)

[0814] An image acquisition means that receives image data from an image acquisition device,

[0815] Image processing means that preprocesses the aforementioned image data and automatically extracts important information,

[0816] A character recognition means that analyzes character information based on the extracted information and converts it into a digital data format,

[0817] A data structuring means that links the aforementioned digital data with purchase information, structures it, and outputs it in tab format,

[0818] A system including a communication means for transmitting the aforementioned tab-formatted data to a remote computing device and automating the return processing.

[0819] (Claim 2)

[0820] The system according to claim 1, characterized in that the communication means converts the tab-formatted data into various data formats and outputs them selectively.

[0821] (Claim 3)

[0822] The system according to claim 1, characterized in that the image acquisition means acquires images by remote operation from a user device and assists with the return procedure.

[0823] "Example 2 of combining an emotion engine"

[0824] (Claim 1)

[0825] An acquisition means for receiving image information from an information processing device that acquires images,

[0826] An analysis means for preprocessing the aforementioned image information and extracting a specific portion of the target information,

[0827] A recognition means that analyzes textual information from the image of the extracted target information and converts it into a data format,

[0828] A means for organizing the character information converted to the aforementioned data format as structured data and outputting it in tabular format,

[0829] An adjustment means for adaptively customizing and outputting the aforementioned tabular data based on the results of an analysis of the user's emotional state,

[0830] A system including a transmission means for transmitting the adjusted data to a remote information processing device.

[0831] (Claim 2)

[0832] The system according to claim 1, characterized in that the transmission means converts the adjusted data into a plurality of data formats and outputs them selectively.

[0833] (Claim 3)

[0834] The system according to claim 1, characterized in that the acquisition means acquires images by remote operation from a user device.

[0835] "Application example 2 when combining with an emotional engine"

[0836] (Claim 1)

[0837] Means for receiving image data from an image acquisition device,

[0838] A processing means for preprocessing the aforementioned image data and extracting a specific part of the object,

[0839] A recognition means that analyzes textual information from the image of the extracted object and converts it into an information format,

[0840] A structuring means that organizes the character information converted into the aforementioned information format as structured information and outputs it in tabular format,

[0841] An analysis means for analyzing the user's emotions based on the aforementioned structured information,

[0842] A presentation means that presents user-specific information based on the aforementioned analysis results,

[0843] A system including communication means for transmitting the aforementioned tabular information to a remote device.

[0844] (Claim 2)

[0845] The system according to claim 1, characterized in that the communication means converts the tabular information into a plurality of information formats and outputs them selectively.

[0846] (Claim 3)

[0847] The system according to claim 1, characterized in that the image acquisition means acquires images by remote operation from a user device. [Explanation of symbols]

[0848] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. An image acquisition means that receives image data from an image acquisition device, Image processing means that preprocesses the aforementioned image data and extracts a specific part of the object, A character recognition means that analyzes character information from the image of the extracted object and converts it into a data format, A data structuring means that organizes the character information converted to the aforementioned data format as structured data and outputs it in tabular format, A system including a communication means for transmitting the aforementioned tabular data to a remote device.

2. The system according to claim 1, characterized in that the communication means converts the tabular data into a plurality of data formats and outputs them selectively.

3. The system according to claim 1, characterized in that the image acquisition means acquires images by remote operation from a user device.

Images