system

Abstract

We provide the system. [Solution] A means for acquiring data including explanatory text and image data, A means for analyzing data and image data containing the above-mentioned explanatory text, and for extracting textual information from the image data, A means for storing extracted text information and explanatory text in a database, A means of searching for data stored in a database based on a user request and sending the retrieved data to the user, A means of displaying data sent to the user, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance that responds to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern museum exhibitions, there are many people who have difficulty accessing due to economic, time, language, and physical handicaps. As a result, these people are restricted from accessing information about exhibits with high accuracy and reliability. Also, there is a problem of a gap in the opportunity to satisfy intellectual curiosity because there is knowledge and cultural information that can only be obtained by visiting the site. Furthermore, it is difficult to quickly and accurately check detailed information about exhibits with existing information-providing means.

Means for Solving the Problems

[0005] To solve the above problems, the present invention provides a system that includes means for acquiring data including descriptive text and image data, means for analyzing the data including the descriptive text and image data and extracting character information from the image data, means for storing the extracted character information and descriptive text in a database, means for searching the stored data from the database based on a user request and transmitting the acquired data to the user, and means for displaying the data transmitted to the user. Furthermore, by including means for using optical character recognition (OCR) technology to extract character information from image data and means for assigning a unique identifier to each data item in order to accurately identify and search the acquired data, efficient and reliable information provision becomes possible. As a result, people with economic, temporal, linguistic, or physical disabilities will be able to quickly and accurately obtain information about museum exhibits.

[0006] "Data including explanatory text" refers to documentary data that includes explanations and descriptions of museum exhibits.

[0007] "Image data" refers to digital data containing visual information, such as photographs and illustrations of exhibits.

[0008] "Means of acquisition" refers to technical methods and devices for collecting data, including descriptive text, and image data, and incorporating them into a system.

[0009] "Means of analysis" refers to the processes and techniques for analyzing acquired data, including explanatory text, and image data to extract specific information.

[0010] "Means for extracting textual information" refers to methods and techniques for identifying and extracting strings of characters or text data from image data, and converting them into a freely readable format.

[0011] "Means of storing in a database" refers to the technologies and methods for storing the analyzed and extracted information in digital storage in a format that the system can later access.

[0012] "Request-based search methods" refer to the processes and techniques for searching and extracting specific information from a database in response to user requests.

[0013] "Means of transmission to the user" refers to the processes and technologies used to deliver retrieved information to the user's device via the network.

[0014] Optical Character Recognition (OCR) is a technology that automatically identifies character information from image data and extracts it as text data.

[0015] A "unique identifier" is a method for streamlining data management and retrieval by assigning a unique identification code to each data entry.

[0016] "Means of display" refers to processes and technologies that visually display transmitted information on the user's device, allowing the user to confirm its content. [Brief explanation of the drawing]

[0017] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7]It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0018] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0021] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0022] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0023] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0025] [First Embodiment]

[0026] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0027] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0030] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0033] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0037] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0038] This invention relates to a system that collects data, including explanatory text, and image data from museums and exhibition facilities, analyzes them, and provides them in a user-accessible format. This system operates with the server, terminal, and user elements working in coordination with each other as follows.

[0039] 1. Data Collection

[0040] The server retrieves descriptive text and image data about the exhibits using the museum's Web API and other data provision methods.

[0041] For example, the server retrieves a list of exhibits and their details by calling the museum's API, GET / api / exhibits.

[0042] 2. Data Analysis and Text Conversion

[0043] The server analyzes the acquired image data of the exhibits and extracts text information from the images using optical character recognition (OCR) technology.

[0044] For example, extract the textual information "History of the Pharaohs" from images of exhibits at the "Ancient Egypt Exhibition."

[0045] 3. Saving to the database

[0046] The server stores the analyzed text information and explanatory data, along with a unique identifier, in a database. This allows for a quick response to user search requests.

[0047] As an example, data on "History of the Pharaohs" associated with ID "AE001" will be saved to the database.

[0048] 4. Processing user requests

[0049] Users use their devices to request information about specific exhibits. For example, a request is sent to the server by accessing a specific URL through the device's browser.

[0050] The server receives this request and searches for the relevant data in the database.

[0051] 5. Data transmission and display

[0052] The server sends the searched data to the user's device. The transmitted data includes descriptions, text information, and related images.

[0053] The terminal visually displays the received data to the user. The user can then review the displayed information and understand the details of the exhibits.

[0054] Specific example

[0055] 1. Examples of data collection

[0056] The server calls the API GET / api / exhibits to retrieve information about the exhibits. For example, information about the "Ancient Egypt Exhibition" is returned as a response. This response includes the exhibit ID, name, description, and image URL.

[0057] The server uses the image URL to download images of the exhibits.

[0058] 2. Examples of data analysis and text conversion

[0059] The server performs OCR processing on the downloaded image. For example, it converts the text information "History of the Pharaohs" within the image into text.

[0060] 3. Example of saving to a database

[0061] The textual information titled "History of the Pharaohs" will be stored in the database, associated with exhibit ID "AE001." This will allow for quick access during future searches.

[0062] 4. Example of processing user requests

[0063] The user sends a request to the server by accessing a URL containing the specific exhibit ID "AE001" from their device's browser.

[0064] 5. Examples of data transmission and display

[0065] The server searches the database for information related to "AE001" and sends it to the user's terminal. The terminal then displays the data, including descriptions and images, to the user.

[0066] Thus, the system of the present invention efficiently collects, analyzes, and stores detailed information about exhibits, making it readily accessible to users, thereby ensuring that information about exhibits is provided fairly to people who have difficulty visiting in person.

[0067] The following describes the processing flow.

[0068] Step 1:

[0069] Data collection (server)

[0070] The server sends a GET request to the museum's Web API, named / api / exhibits, to retrieve a list of exhibits and detailed information about them.

[0071] The received response includes the exhibit ID, name, description, and image URL.

[0072] For example, retrieve {"id": "123", "name": "Ancient Egypt Exhibition", "description": "History of the Pharaohs", "image_url": "https: / / example.com / image1.jpg"}.

[0073] Step 2:

[0074] Image data acquisition (server)

[0075] The server uses the image URL in the response to send a GET / image1.jpg request to download the image data.

[0076] The image data file image1.jpg is retrieved and temporarily saved for later analysis.

[0077] Step 3:

[0078] OCR processing (server)

[0079] The server begins analyzing the acquired image data using optical character recognition (OCR) technology.

[0080] Extract text information from an image and save it as text data. Example: Extract the text "History of the Pharaohs" from the image image1.jpg.

[0081] Step 4:

[0082] Saving to the database (server)

[0083] The server stores the extracted text information and explanatory data, along with a unique identifier, in a database.

[0084] Example: Save the data for "History of the Pharaohs" associated with ID "AE001" to the database in the following format:

[0085] SQL

[0086] INSERT INTO exhibits (id, name, description, ocr_text, image_url)

[0087] VALUES ('AE001', 'Ancient Egypt Exhibition', 'History of the Pharaohs', 'History of the Pharaohs', 'https: / / example.com / image1.jpg');

[0088] Step 5:

[0089] Processing user requests (user, device, server)

[0090] The user uses a terminal to request information about a specific exhibit. For example, they might send a request to GET / api / exhibits / AE001.

[0091] The terminal sends a request to the server, and the server receives this request.

[0092] Step 6:

[0093] Execution of database queries (server)

[0094] The server searches the database for information on the relevant exhibit based on the received request.

[0095] Example: Execute an SQL query to retrieve exhibit data for ID "AE001":

[0096] SQL

[0097] SELECT FROM exhibits WHERE id = 'AE001';

[0098] The acquired data is formatted and converted into a format that can be sent back to the user's terminal.

[0099] Step 7:

[0100] Data transmission (server, terminal)

[0101] The server sends the formatted data to the user's terminal.

[0102] example:

[0103] json

[0104] {

[0105] "id": "AE001",

[0106] "name": "Ancient Egypt Exhibition",

[0107] "description": "History of the Pharaohs",

[0108] "ocr_text": "History of the Pharaohs",

[0109] "image_url": "https: / / example.com / image1.jpg"

[0110] }

[0111] Step 8:

[0112] Data display (terminal, user)

[0113] The terminal processes the received data to display it visually.

[0114] Users can view the explanatory text, OCR-generated text information, and related images displayed on the device screen.

[0115] This process allows users to efficiently and quickly obtain detailed information about exhibits without having to physically visit the museum.

[0116] (Example 1)

[0117] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0118] To provide timely and accurate information about exhibits in museums and exhibition facilities, a system is needed to effectively collect, analyze, and store explanatory texts and image data. However, conventional methods have low accuracy in analyzing collected data, making it difficult to respond quickly to user requests. Furthermore, there is a lack of means to fairly provide information about exhibits to people who have difficulty visiting in person. Therefore, the development of a system that enables efficient and highly accurate data collection, analysis, storage, and provision is necessary.

[0119] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0120] In this invention, the server includes means for acquiring data and image data, including descriptive text related to museums and exhibition facilities; means for extracting character information from image data using optical character recognition technology; means for storing the extracted character information and descriptive text in a database along with a unique identifier; means for searching the database for stored data based on a user request and transmitting the acquired data to the user; and means for displaying the data transmitted to the user. This enables the efficient collection, analysis, and storage of information on exhibits in museums and exhibition facilities, allowing users to access it quickly and providing information fairly to people who have difficulty visiting.

[0121] A museum is a facility for collecting, preserving, and displaying historical, cultural, and artistic exhibits.

[0122] An "exhibition facility" is a place where people can view exhibits for educational or entertainment purposes.

[0123] An "explanatory text" is a piece of writing that provides information and explanations about an exhibit.

[0124] "Data" refers to a collection of information or signals, and in this invention, it includes explanatory text and image data.

[0125] "Image data" refers to digital data that includes visual information such as photographs and illustrations of exhibits.

[0126] "Means of acquisition" refers to the methods and techniques used to collect the necessary data.

[0127] "Means of analysis" refers to methods and techniques for processing acquired data and extracting necessary information.

[0128] "Optical character recognition technology" is a technology that converts characters in an image into digital text.

[0129] "Textual information" refers to characters and sentences that are represented as digital text.

[0130] A "unique identifier" is a unique number or code used to independently identify data.

[0131] A "database" is a system for efficiently storing and managing large amounts of data, and for searching and extracting it as needed.

[0132] "Means of storage" refers to methods and technologies for recording data in storage devices such as databases.

[0133] A "request" refers to the operation or content of a request that a user makes to a system to request information.

[0134] "Means of searching" refers to methods and techniques for finding data within a database.

[0135] "Means of transmission" refers to the methods and technologies used to send data to a specified destination.

[0136] "Means of display" refers to methods and technologies for visually showing data on a device.

[0137] A "user" refers to an individual or organization that uses this system to access information.

[0138] This invention is a system that collects and analyzes explanatory texts and image data of exhibits in museums and exhibition facilities, and provides them in a user-accessible format. This system operates with the server, terminal, and user elements working in coordination with each other. Specific embodiments are described below.

[0139] Data collection methods

[0140] Server operation

[0141] The server calls Web APIs provided by museums and exhibition facilities to retrieve descriptive text and image data about the exhibits. Specifically, the server sends HTTP requests and receives responses in JSON format. The hardware used for this is a cloud service (e.g., AWS® EC2). The software used is the Python language and an HTTP request library (e.g., requests).

[0142] Data analysis and text format

[0143] Server operation

[0144] The server analyzes the acquired image data and extracts text information from the image using optical character recognition (OCR) technology. This process utilizes APIs such as Google® Cloud Vision. The server sends the image data to this API and organizes the returned text data.

[0145] Method of saving to the database

[0146] Server operation

[0147] The server stores the parsed text information and descriptions along with a unique identifier in a database. MySQL® or PostgreSQL are used for this database. The server inserts the text data and exhibit IDs into the database tables.

[0148] User request processing methods

[0149] User actions

[0150] The user requests information about a specific exhibit using their device's browser. Specifically, they access a URL containing the ID of the specific exhibit. For example, they might access "http: / / example.com / exhibit?ID=AE001".

[0151] Server operation

[0152] The server receives a user request and searches for relevant data in the database. It executes an SQL query (e.g., "SELECT FROM exhibits WHERE id='AE001'") to retrieve the search results.

[0153] Data transmission and display methods

[0154] Server operation

[0155] The server sends the search results to the user's device in JSON format. The transmitted data includes descriptions of the exhibits, extracted text information, and associated images.

[0156] Terminal operation

[0157] The terminal analyzes the received data and generates HTML for display. The terminal displays the generated HTML in a browser, allowing the user to view detailed information about the exhibits.

[0158] Specific example

[0159] 1. Examples of data collection

[0160] The server retrieves information about the exhibits by calling "GET / api / exhibits". For example, information about the "Ancient Egypt Exhibition" is returned as a response. This response includes the exhibit ID, name, description, and image URL.

[0161] 2. Examples of data analysis and text conversion

[0162] The server performs OCR processing on the downloaded image, converting the text information "History of the Pharaohs" into text.

[0163] 3. Example of saving to a database

[0164] The textual information titled "History of the Pharaohs" will be stored in the database, associated with exhibit ID "AE001." This will allow for quick access during future searches.

[0165] 4. Example of processing user requests

[0166] The user accesses a URL containing the specific exhibit ID "AE001" from their device's browser and sends a request to the server.

[0167] 5. Examples of data transmission and display

[0168] The server searches the database for information related to "AE001" and sends it to the user's terminal. The terminal then displays the data, including descriptions and images, to the user.

[0169] Example of a prompt

[0170] Prompt: "Explain the important exhibits in the Ancient Egypt section, especially those related to the history of the pharaohs."

[0171] AI model response: "Information related to the history of the pharaohs shows how the kings of ancient Egypt ruled. Exhibit ID 'AE001' contains this information in detail."

[0172] The above describes a specific embodiment for carrying out this invention. This enables efficient and accurate collection, analysis, and storage of exhibit information, and allows users to access it quickly.

[0173] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0174] Step 1: Data Collection

[0175] Server operation

[0176] The server calls the Web API of the museum or exhibition facility to retrieve descriptive text and image data about the exhibits. First, the server sends an HTTP request to the API endpoint (e.g., "GET / api / exhibits"). The response is returned in JSON format. This JSON data includes the exhibit's ID, name, description, and image URL.

[0177] Input: API endpoint URL (e.g., "GET / api / exhibits")

[0178] Output: Exhibit data in JSON format

[0179] Specific actions:

[0180] The server sends an HTTP request: "GET / api / exhibits"

[0181] The server receives a response: data in JSON format.

[0182] The server extracts the exhibit's ID, name, description, and image URL from the response.

[0183] Step 2: Data Analysis and Text Conversion

[0184] Server operation

[0185] The server analyzes the acquired image data and extracts text information from the image using optical character recognition (OCR) technology. Using an OCR service such as the Google Cloud Vision API, the server sends the image and receives the text data.

[0186] Input: Downloaded image data

[0187] Output: Extracted text data

[0188] Specific actions:

[0189] The server sends image data to the Google Cloud Vision API.

[0190] The API recognizes text within an image and returns the text data.

[0191] The server organizes the returned text data into something like "The History of the Pharaohs."

[0192] Step 3: Saving to the database

[0193] Server operation

[0194] The server stores the parsed text information and explanatory text along with a unique identifier (ID) in a database. MySQL or PostgreSQL are used as the database system. Data is inserted via SQL queries.

[0195] Input: Parsed text data, unique identifier, description

[0196] Output: Records stored in the database

[0197] Specific actions:

[0198] The server inserts text data and exhibit IDs into the database using an SQL query.

[0199] Example: "INSERT INTO exhibits (id, name, description, text_data) VALUES ('AE001', 'History of the Pharaohs', 'Details about the Pharaohs', 'Text data in the image')"

[0200] Step 4: Processing user requests

[0201] User actions

[0202] The user requests information about a specific exhibit using their device's browser. Specifically, they access a URL that includes the exhibit ID (e.g., "http: / / example.com / exhibit?ID=AE001").

[0203] Server operation

[0204] The server parses the received request and searches for relevant data in the database. It then executes an SQL query to retrieve information about the requested exhibit.

[0205] Input: User request (URL)

[0206] Output: Exhibit data retrieved from the database

[0207] Specific actions:

[0208] The user accesses "http: / / example.com / exhibit?ID=AE001" in their browser.

[0209] The server receives the request and executes the SQL query "SELECT FROM exhibits WHERE id='AE001'".

[0210] The server retrieves the query results.

[0211] Step 5: Send and display data

[0212] Server operation

[0213] The server sends the search results to the user's device in JSON format. The transmitted data includes descriptions of the exhibits, extracted text information, and related images.

[0214] Terminal operation

[0215] The terminal analyzes the received data and generates HTML for display. The terminal then displays the generated HTML in a browser, allowing the user to view detailed information about the exhibits.

[0216] Input: Exhibit data (in JSON format) retrieved from the database.

[0217] Output: Exhibit information displayed visually to the user

[0218] Specific actions:

[0219] The server sends the search results to the terminal in JSON format.

[0220] The terminal parses the received JSON data and generates HTML.

[0221] The device displays the generated HTML in a browser, allowing the user to view detailed information about the exhibits.

[0222] The above outlines the specific processing steps of the system. This process allows users to efficiently access information about the exhibits.

[0223] (Application Example 1)

[0224] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0225] Currently, physical stores face the challenge of users not being able to easily obtain detailed product information and review data. In particular, small print on product labels or a lack of detailed information can reduce user purchasing intent. Furthermore, providing product information visually often requires additional effort from the store, making efficient information provision difficult. An efficient system is needed to solve these problems.

[0226] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0227] In this invention, the server includes means for acquiring data including descriptive text and image data; means for using optical character recognition technology to extract character information from the image data; means for storing the extracted character information and descriptive text in a database; means for searching the database for stored data based on a user request and transmitting the acquired data to the user; means for displaying the data transmitted to the user; and means for using a smartphone to acquire product identification information and display product descriptions. This allows users to easily obtain detailed information about products in physical stores and increase their purchasing intent. Furthermore, stores can provide product information efficiently, improving overall operational efficiency.

[0228] "Data including descriptive text" refers to text-based data that contains detailed information about exhibits or products.

[0229] "Image data" refers to data that includes photographs, diagrams, and other visual information of exhibits and products.

[0230] "Optical character recognition technology" is a technology that analyzes character information in an image and extracts it as text data.

[0231] A "database" is an information processing system for centrally storing and managing collected data.

[0232] A "unique identifier" is a unique ID number or code used to distinguish each piece of data from other pieces of data.

[0233] A "user request" is an access request made by a user seeking specific information.

[0234] "Searching" is the process of finding specific data from stored information.

[0235] "Sending" refers to the act of transferring the retrieved data to the user's device.

[0236] "Display" refers to the act of providing data visually to a user's device.

[0237] "Product identification information" refers to information such as barcodes and QR codes (registered trademarks) used to identify specific products.

[0238] A "smartphone" is a mobile phone-type information terminal used by users to obtain and display information.

[0239] To implement this invention, a system is required in which the server, terminal, and user elements work together in coordination. Below, a specific embodiment of this system, a product information provision system in a physical store, will be described.

[0240] Data collection

[0241] The server uses the store's API to retrieve product descriptions and image data. For example, it can retrieve data including product ID, product name, description, price, and image URL from the store's API. Furthermore, it can also collect product images in real time from cameras installed in the store.

[0242] Data analysis and text conversion

[0243] The server uses optical character recognition (OCR) technology to extract text information from collected product images. For example, it can retrieve information such as "organically grown" or "expiration date" from product packaging labels. The pytesseract library is used for this OCR processing.

[0244] Save to database

[0245] The server stores the extracted text information and descriptions in a database along with a unique identifier (product ID). This allows for quick data retrieval later. A database management system such as SQLite is used for the database.

[0246] Processing user requests

[0247] The user uses a smartphone app to scan product identification information, such as a QR code. The scanned information is sent to the server as a request. Based on this request, the server searches its database for the relevant product data.

[0248] Data transmission and display

[0249] The server sends the searched data to the user's smartphone. The transmitted data includes product names, descriptions, prices, images, ingredient information, and reviews. The user can visually confirm this information on their smartphone screen.

[0250] Specific example

[0251] Specific examples of data collection

[0252] The data retrieved from the store API is in the following format:

[0253] Product ID: "PRD001"

[0254] Product name: "Organic Green Tea"

[0255] Description: "100% organically grown, pesticide-free tea"

[0256] Price: 500 yen

[0257] Image URL: "https: / / store.com / images / greentea.jpg"

[0258] Specific examples of analysis using OCR

[0259] The server extracts the following text information from the product image:

[0260] "Organic cultivation"

[0261] "Best before date: December 2025"

[0262] Example of a prompt

[0263] Data collection:

[0264] response = requests.get('https: / / store-api.com / products')

[0265] OCR analysis:

[0266] text = pytesseract.image_to_string(Image.open('path / to / image.jpg'), lang='jpn')

[0267] In this way, the system according to this invention provides an environment in which users can easily obtain detailed information about products in physical stores, and enables stores to provide information efficiently.

[0268] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0269] Step 1:

[0270] Data collection

[0271] The server sends a request to the store's API to retrieve product descriptions and image data. Specifically, the server sends an HTTP GET request to the API endpoint and receives data in the response, including product ID, product name, description, price, and image URL. Input: API request, Output: Product data JSON.

[0272] Step 2:

[0273] Download image data

[0274] The server downloads product images based on the image URLs in the retrieved product data. Specifically, it sends an HTTP GET request to the image URL for each product and retrieves the image file as a response. Input: Image URL, Output: Image file.

[0275] Step 3:

[0276] Extraction of text information using OCR

[0277] The server extracts text information from downloaded image files using Optical Character Recognition (OCR) technology. Specifically, it uses the pytesseract library to extract text from image files. This text serves as supplementary information about the product. Input: Image file, Output: Extracted text.

[0278] Step 4:

[0279] Save to database

[0280] The server stores the extracted character information and the obtained product data in the database. Each data is assigned a unique identifier (product ID) and is managed efficiently. Specifically, the data is registered in the SQLite database using the INSERT statement. Input: product data and extracted text, Output: database records.

[0281] Step 5:

[0282] Scanning of product identification information by the user

[0283] The user uses the smartphone app to scan the identification information such as the product QR code in the store. Specifically, the camera of the smartphone is used to read the QR code, and the read data is displayed within the app. Input: QR code, Output: product identification information.

[0284] Step 6:

[0285] Sending of user request

[0286] The user sends the product identification information to the server as a request through the app. Specifically, the smartphone app sends an HTTP GET request containing the product ID to the server. Input: product identification information, Output: request sending.

[0287] Step 7:

[0288] Search from the database

[0289] The server searches the database based on the product identification information received from the user and obtains the relevant product information. Specifically, a SELECT statement is used in the database for the search using the product ID. Input: product identification information, Output: product data.

[0290] Step 8:

[0291] Sending of product data to the user terminal

[0292] The server sends the retrieved product data to the user's smartphone. Specifically, it sends the retrieved data as an HTTP response in JSON format. Input: Product data, Output: JSON response.

[0293] Step 9:

[0294] Displaying product information

[0295] The user visually displays product information received via a smartphone app. Specifically, the app displays product name, description, price, ingredient information, reviews, etc., on the screen. Input: JSON response, Output: Display of product information.

[0296] In this way, the server, terminal, and user work together at each step, making it possible for users to easily obtain product information in physical stores. This allows users to make purchasing decisions based on more detailed information, and stores can provide information more efficiently.

[0297] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0298] This invention combines a system that collects data including explanatory text and image data from museums and exhibition facilities, analyzes them, and provides them in a user-accessible format with an emotion engine that recognizes user emotions. This emotion engine makes it possible to dynamically optimize the information displayed according to the user's emotional state. Furthermore, it can learn from emotional data and provide information tailored to the user's preferences.

[0299] 1. Data Collection

[0300] The server uses the museum's Web API or other data - providing means to obtain the explanatory text and image data related to the exhibits.

[0301] For example, the server calls the API GET / api / exhibits provided by the museum to obtain the exhibit list and its detailed information.

[0302] 2. Acquisition of Image Data

[0303] The server uses the image URL in the response to send a request for GET / image1.jpg and downloads the image data.

[0304] The server acquires the image data image1.jpg and temporarily stores it for later analysis.

[0305] 3. OCR Processing

[0306] The server starts analyzing the acquired image data using optical character recognition technology (OCR).

[0307] Extract the character information in the image and save it as text data. Example: Extract the text "History of Pharaohs" from the image image1.jpg.

[0308] 4. Saving to the Database

[0309] The server saves the data including the analyzed character information and the explanatory text to the database together with a unique identifier.

[0310] Example: Save the data of "History of Pharaohs" associated with the ID "AE001" to the database in the following format:

[0311] sql

[0312] INSERT INTO exhibits (id, name, description, ocr_text, image_url)

[0313] VALUES ('AE001', 'Ancient Egypt Exhibition', 'History of the Pharaohs', 'History of the Pharaohs', 'https: / / example.com / image1.jpg');

[0314] 5. Processing user requests

[0315] Users use their devices to request information about specific exhibits. For example, a request is sent to the server by accessing a specific URL through the device's browser.

[0316] The terminal sends a request to the server, and the server receives this request.

[0317] 6. Executing database queries

[0318] The server searches the database for information on the relevant exhibit based on the received request.

[0319] Example: Execute an SQL query to retrieve exhibit data for ID "AE001":

[0320] SQL

[0321] SELECT FROM exhibits WHERE id = 'AE001';

[0322] The acquired data is formatted and converted into a format that can be sent back to the user's terminal.

[0323] 7. Emotion recognition by an emotion engine

[0324] The server uses cameras and sensors installed on the user's device to recognize the user's emotional state in real time.

[0325] The emotion engine uses facial recognition and voice analysis technologies to extract emotions from the user's facial expressions and speech.

[0326] The recognized emotional state (e.g., joy, interest, surprise, confusion, etc.) is stored as data.

[0327] 8. Data transmission and display

[0328] The server dynamically optimizes the data it displays based on the emotional state recognized by the emotion engine.

[0329] The optimized data, including the exhibit descriptions, OCR text information, and related images, is sent to the user's device.

[0330] The terminal processes the received data to display it visually and then provides it to the user.

[0331] 9. Learning from user sentiment data

[0332] The server stores user emotion data recognized by the emotion engine and learns the user's preferences and interests.

[0333] The learned data will be used to provide information in the future. For example, if there is new information related to exhibits that the user has shown interest in in the past, the system will be adjusted to prioritize displaying that information.

[0334] Specific example

[0335] 1. Examples of data collection

[0336] The server calls the API GET / api / exhibits to retrieve information about the exhibits. For example, information about the "Ancient Egypt Exhibition" is returned as a response. This response includes the exhibit ID, name, description, and image URL.

[0337] The server uses the image URL to download images of the exhibits.

[0338] 2. Examples of data analysis and text conversion

[0339] The server performs OCR processing on the downloaded image. For example, it converts the text information "History of the Pharaohs" within the image into text.

[0340] 3. Examples of optimization using an emotion engine

[0341] When a user requests information about "AE001" and the device receives data from the server, the server analyzes the user's camera footage and recognizes that the user is in an "excited" state.

[0342] The server sends optimized data, including more detailed information and additional images, to users in an "excited" state.

[0343] 4. Example of data display

[0344] The device visually displays and provides optimized data to the user. For example, it might show a detailed description of "The History of the Pharaohs" along with several related images.

[0345] In this way, the system of the present invention can efficiently collect, analyze, and store detailed information about exhibits, making it readily accessible to users, and can also optimize the information according to the user's emotional state, providing a personalized experience.

[0346] The following describes the processing flow.

[0347] Step 1:

[0348] Data collection (server)

[0349] The server sends a GET request to the museum's Web API, named / api / exhibits, to retrieve a list of exhibits and detailed information about them.

[0350] The received response includes the exhibit ID, name, description, and image URL.

[0351] For example, retrieve {"id": "123", "name": "Ancient Egypt Exhibition", "description": "History of the Pharaohs", "image_url": "https: / / example.com / image1.jpg"}.

[0352] Step 2:

[0353] Image data acquisition (server)

[0354] The server uses the image URL in the response to send a GET / image1.jpg request to download the image data.

[0355] The image data file image1.jpg is retrieved and temporarily saved for later analysis.

[0356] Step 3:

[0357] OCR processing (server)

[0358] The server begins analyzing the acquired image data using optical character recognition (OCR) technology.

[0359] This process extracts text information from an image and saves it as text data. For example, it extracts the text "History of the Pharaohs" from the image image1.jpg.

[0360] Step 4:

[0361] Saving to the database (server)

[0362] The server stores the analyzed text information and explanatory data, along with a unique identifier, in a database.

[0363] As an example, the data for "History of the Pharaohs" associated with ID "AE001" will be saved to the database in the following format:

[0364] SQL

[0365] INSERT INTO exhibits (id, name, description, ocr_text, image_url)

[0366] VALUES ('AE001', 'Ancient Egypt Exhibition', 'History of the Pharaohs', 'History of the Pharaohs', 'https: / / example.com / image1.jpg');

[0367] Step 5:

[0368] Processing user requests (user, device, server)

[0369] The user uses their device to request information about a specific exhibit. For example, they might send a request like GET / api / exhibits / AE001 from their device's browser.

[0370] The server receives this request and prepares the corresponding data.

[0371] Step 6:

[0372] Execution of database queries (server)

[0373] Based on the request, the server searches the database for information about the relevant exhibit.

[0374] For example, execute an SQL query to retrieve exhibit data for ID "AE001":

[0375] SQL

[0376] SELECT FROM exhibits WHERE id = 'AE001';

[0377] The acquired data is formatted and converted into a format that can be sent back to the user's terminal.

[0378] Step 7:

[0379] Emotion recognition using an emotion engine (server, terminal)

[0380] The device uses cameras and sensors to capture the user's facial expressions and voice, and sends this information to the emotion engine.

[0381] The server uses an emotion engine to analyze and recognize the user's emotional state in real time.

[0382] For example, the emotion engine recognizes that a user is in a state of "excitement" after viewing the exhibit's description.

[0383] Step 8:

[0384] Data transmission and display (server, terminal, user)

[0385] The server dynamically optimizes the data it displays based on the recognized emotional state.

[0386] The optimized data, including the exhibit descriptions, OCR text information, and related images, is sent to the user's device.

[0387] The terminal visually displays the received data, allowing users to view detailed information about the exhibits.

[0388] Step 9:

[0389] Learning user sentiment data (server)

[0390] The server stores user emotion data recognized by the emotion engine and learns the user's preferences and interests.

[0391] For example, the system could be adjusted to prioritize displaying information related to exhibits the user has previously shown interest in during subsequent requests.

[0392] This process allows the system to provide information that responds to the user's real-time emotions, delivering a personalized experience.

[0393] (Example 2)

[0394] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0395] Traditional museum and exhibition information systems have faced challenges in maintaining user interest because the information provided is not dynamically optimized according to the user's emotional state or individual interests. Furthermore, the lack of consideration for user emotional states hinders the quality of understanding and experience of exhibits. Additionally, inefficient management and retrieval of acquired data, resulting in a mix of non-uniquely identifiable data, makes it difficult to quickly provide necessary information.

[0396] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0397] In this invention, the server includes means for acquiring data including descriptive text and image data; means for extracting character information from the acquired image data using optical character recognition technology; means for storing the extracted character information and descriptive text in a database; means for searching the database for stored data based on a user request and transmitting the acquired data to the user; means for recognizing the user's emotional state; means for dynamically optimizing the data to be displayed based on the recognized emotional state; and means for visually displaying the data on the user terminal. This enables personalized information provision according to the user's emotional state and interests, thereby improving the quality of the user experience. Furthermore, it enables efficient data management and retrieval, and allows for the rapid provision of uniquely identified data.

[0398] "Data including descriptive text" refers to data that contains descriptive information in text format about exhibits and related information.

[0399] "Image data" refers to data in image format that visually represents exhibits and related information.

[0400] "Optical character recognition technology" is a technology for extracting character information from image data, and is also known as OCR (Optical Character Recognition).

[0401] "Character information" refers to information in text format extracted using optical character recognition technology.

[0402] A "database" is a digital system for efficiently storing, managing, and retrieving acquired and analyzed data.

[0403] A "user request" is a request that a user sends to the system seeking specific information.

[0404] "User emotional state" refers to information about the user's current mental and emotional condition, and is recognized from facial expressions, voice analysis, and other sources.

[0405] "Dynamic optimization" means adjusting the information displayed in real time according to the user's current state and situation, and providing it in the most optimal form.

[0406] "Visually displaying" means displaying information graphically on the user's device, and it means providing images and text in a format that can be visually confirmed.

[0407] A "unique identifier" is a unique identifier assigned to each piece of data to distinguish it from other pieces of data.

[0408] This invention relates to a system for collecting, analyzing, and storing detailed information about exhibits in museums and exhibition facilities, and providing personalized information to users. This system acquires data including descriptive text and image data, analyzes them, and provides them in a user-accessible format. Furthermore, it can recognize user emotions and dynamically optimize the information. Specific embodiments are described below.

[0409] Hardware and software to be used

[0410] This system consists of three main elements: a server, terminals, and users. The server is a computer system with a high-performance processor and sufficient memory, which collects and analyzes data. It also uses an SQL-based database (e.g., PostgreSQL) as its database system. For image data analysis, it uses Tesseract, a widely used optical character recognition (OCR) technology. For emotion recognition, it uses an emotion analysis module called EmotionEngine. User terminals include smartphones, tablets, and personal computers equipped with cameras and sensors.

[0411] Data collection and analysis

[0412] The server calls the museum's API to retrieve descriptive text and image data about the exhibits. The retrieved image data is converted into text information using optical character recognition (OCR) technology. For example, it calls the API GET / api / exhibits, obtains the image URL from the response, accesses the corresponding URL to download the image, and extracts text information from that image. Through this process, the text data "History of the Pharaohs" is extracted from the image image1.jpg.

[0413] Data storage

[0414] The server stores the extracted text information and descriptions in the database, associating them with a unique identifier (e.g., ID "AE001"). For example, data related to the "Ancient Egypt Exhibition" is stored in the database in the following format:

[0415] Exhibit ID: AE001

[0416] Exhibition Title: Ancient Egypt Exhibition

[0417] Description: History of the Pharaohs

[0418] OCR text: History of the Pharaohs

[0419] Image URL: https: / / example.com / image1.jpg

[0420] Processing and optimizing user requests

[0421] Users request information about specific exhibits using their devices. For example, accessing a specific URL from a browser sends a request to the server. The server receives this request, searches its database for information about the relevant exhibit, and sends it back to the user. Furthermore, the system recognizes the user's emotional state in real time through the device's camera and sensors, and extracts it using an emotion engine. Based on the recognized emotional state (e.g., "excited"), the displayed data is dynamically optimized. For example, if the user is in an "excited" state, data containing more detailed information and additional images is sent to the user's device. The device then visually displays the optimized information based on this.

[0422] Learning user sentiment data

[0423] The server stores user emotion data recognized by the emotion engine and learns the user's preferences and interests. This learned data is used to provide information in the future. For example, if there is new information related to an exhibit that the user has shown interest in in the past, that information will be displayed preferentially.

[0424] In this way, the system of the present invention can collect, analyze, and store detailed information about exhibits, making it readily accessible to users, and can also optimize the information according to the user's emotional state to provide a personalized experience.

[0425] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0426] Step 1:

[0427] Data collection

[0428] The server calls the museum's Web API to retrieve descriptive text and image data about the exhibits.

[0429] Input: API call endpoint (e.g., GET / api / exhibits)

[0430] Data processing: Analyze the JSON response data obtained from the API and extract a list of exhibits.

[0431] Output: Exhibit list (e.g., exhibit ID, name, description, image URL, etc.)

[0432] Step 2:

[0433] Acquisition of image data

[0434] The server extracts image URLs from the list of exhibits and uses those URLs to download the image data.

[0435] Input: Image URL from the exhibit list (e.g., https: / / example.com / image1.jpg)

[0436] Data processing: Send an HTTP request and receive the corresponding image data.

[0437] Output: Image data file (e.g., image1.jpg)

[0438] Step 3:

[0439] OCR processing

[0440] The server applies optical character recognition (OCR) technology to the downloaded image data to extract text information.

[0441] Input: Image data file (e.g., image1.jpg)

[0442] Data processing: Analyze the text in the image using OCR software (e.g., Tesseract).

[0443] Output: Extracted text data (e.g., "History of the Pharaohs")

[0444] Step 4:

[0445] Save to database

[0446] The server stores the extracted text information and descriptions in a database along with a unique identifier.

[0447] Input: Exhibit ID, Name, Description, OCR text, Image URL

[0448] Data processing: Execute SQL queries to insert information into the database.

[0449] Output: Database entry where the data is stored (e.g., information associated with ID "AE001")

[0450] Step 5:

[0451] Processing user requests

[0452] Users use their devices to request information about specific exhibits.

[0453] Input: Request URL sent by the user from their browser (e.g., https: / / museum-website.com / exhibit?id=AE001)

[0454] Data processing: The terminal sends a request to the server, and the server receives the request.

[0455] Output: Request information forwarded to the server

[0456] Step 6:

[0457] Execution of database queries

[0458] Based on the received request, the server searches the database for information about the relevant exhibit.

[0459] Input: Exhibit ID in the request (e.g., AE001)

[0460] Data processing: Execute SQL queries to search for the relevant data.

[0461] Output: Searched exhibit information (e.g., exhibit name, description, OCR text, image URL)

[0462] Step 7:

[0463] Emotion recognition by an emotion engine

[0464] The server uses cameras and sensors installed on the user's device to recognize the user's emotional state in real time.

[0465] Input: User's camera video and audio data

[0466] Data processing: Analyze emotional information using an emotion engine (e.g., EmotionEngine).

[0467] Output: Recognized emotional state (e.g., "excited")

[0468] Step 8:

[0469] Data transmission and display

[0470] The server dynamically optimizes the data displayed based on the recognized emotional state.

[0471] Input: Exhibit information, perceived emotional state

[0472] Data processing: Generate optimized information and send it to the user's terminal.

[0473] Output: Dynamically optimized exhibit information

[0474] Step 9:

[0475] Displaying data

[0476] The terminal visually displays the received data and provides it to the user.

[0477] Input: Optimization information sent from the server

[0478] Data processing: Converting data into a layout and format for display.

[0479] Output: Detailed exhibit information provided to the user visually (e.g., detailed description, images)

[0480] Step 10:

[0481] Learning user sentiment data

[0482] The server stores user emotion data recognized by the emotion engine and learns the user's preferences and interests.

[0483] Input: Recognized emotional state, corresponding exhibit information

[0484] Data processing: Emotional data and exhibit information are stored in a database and analyzed.

[0485] Output: Recommendations based on learned user preferences

[0486] (Application Example 2)

[0487] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0488] Conventional exhibit information systems lacked dynamic data optimization based on user emotional states, resulting in insufficient improvement in the quality of the user experience. Furthermore, they lacked personalized information tailored to user preferences, making it difficult to quickly and accurately provide information of interest to users. Additionally, conventional systems lacked the functionality to understand and adjust displayed content based on real-time user emotional states, thus failing to provide information that met user interests and concerns.

[0489] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0490] In this invention, the server includes means for acquiring data including descriptive text and image data; means for analyzing the aforementioned data including descriptive text and image data and extracting text information from the image data; means for storing the extracted text information and descriptive text in a database; means for searching the database for stored data based on a user request and sending the acquired data to the user; means for displaying the data sent to the user; means for incorporating an emotion engine that analyzes sensor data acquired from a terminal to recognize the user's emotional state; and means for dynamically optimizing the information to be displayed according to the user's emotional state. This makes it possible to read the user's emotional state and provide information that corresponds to it in real time with dynamic optimization. Personalized information provision tailored to the user's interests and concerns can be realized, improving the quality of the user experience.

[0491] A "description" is text data that provides detailed information about an exhibit.

[0492] "Image data" refers to digital image files that include visual representations of exhibits and related items.

[0493] A "database" is an information system for systematically storing and managing descriptive text and image data.

[0494] A "user" or "device" refers to an individual requesting information about an exhibit and the device they use (e.g., a smartphone or tablet).

[0495] A "request" is a request from a user to retrieve information about a specific exhibit from a database.

[0496] An "emotion engine" is a system that recognizes the user's emotional state and dynamically optimizes the information displayed based on that data.

[0497] "Sensor data" refers to data such as the user's facial expressions and voice, acquired from the device's camera, microphone, etc.

[0498] "Dynamic optimization" is a process that adjusts the displayed information according to the user's real-time emotional state.

[0499] "Extraction" is the process of extracting text information from image data.

[0500] "Storage" refers to the process of recording and storing the analyzed data in a database.

[0501] This invention aims to improve the user experience in virtual stores. This system uses an emotion engine to recognize the user's emotional state and provides optimized product information tailored to that emotion. Specifically, the server implements this invention through the following main means.

[0502] The server first retrieves data including descriptions and image data. This data is collected from virtual stores via an API.

[0503] The server uses optical character recognition (OCR) technology to extract text information from the acquired image data. This allows for obtaining detailed text information about the product from the image data.

[0504] The parsed text information and explanatory text are stored in a database. The stored data is associated with a unique identifier, facilitating subsequent searching and retrieval.

[0505] The user requests specific product information using a device (e.g., a smartphone). In response to this request, the server searches the database, retrieves the relevant product data, and sends it to the user's device.

[0506] During this process, sensor data is acquired in real time from the camera and microphone installed in the device, and the emotion engine analyzes the user's emotional state. The emotion engine uses the Google Cloud Vision API and the Hugging Face Transformers library, among others, to recognize emotions from the user's facial expressions and voice.

[0507] The information displayed is dynamically optimized according to the recognized emotional state (e.g., "joy" or "surprise"). For example, if the emotion of "joy" is detected, the system will prioritize displaying new product and promotional information that the user is likely to be interested in.

[0508] To give a concrete example, when a user opens a virtual store app on their smartphone and points their face at the camera, the app's emotion engine recognizes "joy." As a result, the next shopping screen prioritizes displaying new products and promotional information that are of interest to the user.

[0509] Examples of prompt statements are as follows:

[0510] "By analyzing the user's facial expression, if an emotion of 'joy' is detected, we will suggest products that are suitable for that user."

[0511] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0512] Step 1:

[0513] The server retrieves data including descriptions and image data. It uses an API to collect product data from virtual stores. The input is the API endpoint URL, and the output is a list of product descriptions and image URLs.

[0514] Step 2:

[0515] The server extracts text information from the acquired image data using optical character recognition (OCR) technology. Specifically, it analyzes the image data using the Google Cloud Vision API and extracts the text information. The input is image data, and the output is the extracted text data.

[0516] Step 3:

[0517] The server stores the analyzed text information and explanatory text in a database. The database stores data using unique identifiers. Input consists of text information, explanatory text, and a unique identifier, while output is the verification result of the stored data.

[0518] Step 4:

[0519] The user requests specific product information using a terminal. The user's request is sent to the server. The input is the user's request content (such as the product ID), and the output is the result of sending the request to the server.

[0520] Step 5:

[0521] The server searches the database based on the user's request and retrieves the relevant product data. The input is the user's request, and the output is the relevant product data.

[0522] Step 6:

[0523] The server sends the acquired product data to the user's terminal. The input is the acquired product data, and the output is the result of the transmission to the user's terminal.

[0524] Step 7:

[0525] The device acquires sensor data in real time from its built-in camera and microphone, and transmits this data to the emotion engine. The input is the sensor data from the camera and microphone, and the output is the result of the data transmission to the emotion engine.

[0526] Step 8:

[0527] The emotion engine uses the Google Cloud Vision API and the Hugging Face Transformers library to analyze the user's emotional state from their facial expressions and voice. The input is sensor data, and the output is the recognized emotional state.

[0528] Step 9:

[0529] Based on the recognized emotional state, the server dynamically optimizes the information it displays. Specifically, it adjusts product information and suggestions according to the user's emotions. The input is emotional state data, and the output is optimized product information.

[0530] Step 10:

[0531] The server sends optimized product information to the user's terminal, which then displays it visually. The input is the optimized product information, and the output is the visual display result provided to the user.

[0532] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0533] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0534] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0535] [Second Embodiment]

[0536] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0537] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0538] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0539] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0540] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0541] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0542] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0543] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0544] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0545] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0546] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0547] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0548] This invention relates to a system that collects data, including explanatory text, and image data from museums and exhibition facilities, analyzes them, and provides them in a user-accessible format. This system operates with the server, terminal, and user elements working in coordination with each other as follows.

[0549] 1. Data Collection

[0550] The server retrieves descriptive text and image data about the exhibits using the museum's Web API and other data provision methods.

[0551] For example, the server retrieves a list of exhibits and their details by calling the museum's API, GET / api / exhibits.

[0552] 2. Data Analysis and Text Conversion

[0553] The server analyzes the acquired image data of the exhibits and extracts text information from the images using optical character recognition (OCR) technology.

[0554] For example, extract the textual information "History of the Pharaohs" from images of exhibits at the "Ancient Egypt Exhibition."

[0555] 3. Saving to the database

[0556] The server stores the analyzed text information and explanatory data, along with a unique identifier, in a database. This allows for a quick response to user search requests.

[0557] As an example, data on "History of the Pharaohs" associated with ID "AE001" will be saved to the database.

[0558] 4. Processing user requests

[0559] Users use their devices to request information about specific exhibits. For example, a request is sent to the server by accessing a specific URL through the device's browser.

[0560] The server receives this request and searches for the relevant data in the database.

[0561] 5. Data transmission and display

[0562] The server sends the searched data to the user's device. The transmitted data includes descriptions, text information, and related images.

[0563] The terminal visually displays the received data to the user. The user can then review the displayed information and understand the details of the exhibits.

[0564] Specific example

[0565] 1. Examples of data collection

[0566] The server calls the API GET / api / exhibits to retrieve information about the exhibits. For example, information about the "Ancient Egypt Exhibition" is returned as a response. This response includes the exhibit ID, name, description, and image URL.

[0567] The server uses the image URL to download images of the exhibits.

[0568] 2. Examples of data analysis and text conversion

[0569] The server performs OCR processing on the downloaded image. For example, it converts the text information "History of the Pharaohs" within the image into text.

[0570] 3. Example of saving to a database

[0571] The textual information titled "History of the Pharaohs" will be stored in the database, associated with exhibit ID "AE001." This will allow for quick access during future searches.

[0572] 4. Example of processing user requests

[0573] The user sends a request to the server by accessing a URL containing the specific exhibit ID "AE001" from their device's browser.

[0574] 5. Examples of data transmission and display

[0575] The server searches the database for information related to "AE001" and sends it to the user's terminal. The terminal then displays the data, including descriptions and images, to the user.

[0576] Thus, the system of the present invention efficiently collects, analyzes, and stores detailed information about exhibits, making it readily accessible to users, thereby ensuring that information about exhibits is provided fairly to people who have difficulty visiting in person.

[0577] The following describes the processing flow.

[0578] Step 1:

[0579] Data collection (server)

[0580] The server sends a GET request to the museum's Web API, named / api / exhibits, to retrieve a list of exhibits and detailed information about them.

[0581] The received response includes the exhibit ID, name, description, and image URL.

[0582] For example, retrieve {"id": "123", "name": "Ancient Egypt Exhibition", "description": "History of the Pharaohs", "image_url": "https: / / example.com / image1.jpg"}.

[0583] Step 2:

[0584] Image data acquisition (server)

[0585] The server uses the image URL in the response to send a GET / image1.jpg request to download the image data.

[0586] The image data file image1.jpg is retrieved and temporarily saved for later analysis.

[0587] Step 3:

[0588] OCR processing (server)

[0589] The server begins analyzing the acquired image data using optical character recognition (OCR) technology.

[0590] Extract text information from an image and save it as text data. Example: Extract the text "History of the Pharaohs" from the image image1.jpg.

[0591] Step 4:

[0592] Saving to the database (server)

[0593] The server stores the extracted text information and explanatory data, along with a unique identifier, in a database.

[0594] Example: Save the data for "History of the Pharaohs" associated with ID "AE001" to the database in the following format:

[0595] SQL

[0596] INSERT INTO exhibits (id, name, description, ocr_text, image_url)

[0597] VALUES ('AE001', 'Ancient Egypt Exhibition', 'History of the Pharaohs', 'History of the Pharaohs', 'https: / / example.com / image1.jpg');

[0598] Step 5:

[0599] Processing user requests (user, device, server)

[0600] The user uses a terminal to request information about a specific exhibit. For example, they might send a request to GET / api / exhibits / AE001.

[0601] The terminal sends a request to the server, and the server receives this request.

[0602] Step 6:

[0603] Execution of database queries (server)

[0604] The server searches the database for information on the relevant exhibit based on the received request.

[0605] Example: Execute an SQL query to retrieve exhibit data for ID "AE001":

[0606] SQL

[0607] SELECT FROM exhibits WHERE id = 'AE001';

[0608] The acquired data is formatted and converted into a format that can be sent back to the user's terminal.

[0609] Step 7:

[0610] Data transmission (server, terminal)

[0611] The server sends the formatted data to the user's terminal.

[0612] example:

[0613] json

[0614] {

[0615] "id": "AE001",

[0616] "name": "Ancient Egypt Exhibition",

[0617] "description": "History of the Pharaohs",

[0618] "ocr_text": "History of the Pharaohs",

[0619] "image_url": "https: / / example.com / image1.jpg"

[0620] }

[0621] Step 8:

[0622] Data display (terminal, user)

[0623] The terminal processes the received data to display it visually.

[0624] Users can view the explanatory text, OCR-generated text information, and related images displayed on the device screen.

[0625] This process allows users to efficiently and quickly obtain detailed information about exhibits without having to physically visit the museum.

[0626] (Example 1)

[0627] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0628] To provide timely and accurate information about exhibits in museums and exhibition facilities, a system is needed to effectively collect, analyze, and store explanatory texts and image data. However, conventional methods have low accuracy in analyzing collected data, making it difficult to respond quickly to user requests. Furthermore, there is a lack of means to fairly provide information about exhibits to people who have difficulty visiting in person. Therefore, the development of a system that enables efficient and highly accurate data collection, analysis, storage, and provision is necessary.

[0629] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0630] In this invention, the server includes means for acquiring data and image data, including descriptive text related to museums and exhibition facilities; means for extracting character information from image data using optical character recognition technology; means for storing the extracted character information and descriptive text in a database along with a unique identifier; means for searching the database for stored data based on a user request and transmitting the acquired data to the user; and means for displaying the data transmitted to the user. This enables the efficient collection, analysis, and storage of information on exhibits in museums and exhibition facilities, allowing users to access it quickly and providing information fairly to people who have difficulty visiting.

[0631] A museum is a facility for collecting, preserving, and displaying historical, cultural, and artistic exhibits.

[0632] An "exhibition facility" is a place where people can view exhibits for educational or entertainment purposes.

[0633] An "explanatory text" is a piece of writing that provides information and explanations about an exhibit.

[0634] "Data" refers to a collection of information or signals, and in this invention, it includes explanatory text and image data.

[0635] "Image data" refers to digital data that includes visual information such as photographs and illustrations of exhibits.

[0636] "Means of acquisition" refers to the methods and techniques used to collect the necessary data.

[0637] "Means of analysis" refers to methods and techniques for processing acquired data and extracting necessary information.

[0638] "Optical character recognition technology" is a technology that converts characters in an image into digital text.

[0639] "Textual information" refers to characters and sentences that are represented as digital text.

[0640] A "unique identifier" is a unique number or code used to independently identify data.

[0641] A "database" is a system for efficiently storing and managing large amounts of data, and for searching and extracting it as needed.

[0642] "Means of storage" refers to methods and technologies for recording data in storage devices such as databases.

[0643] A "request" refers to the operation or content of a request that a user makes to a system to request information.

[0644] "Means of searching" refers to methods and techniques for finding data within a database.

[0645] "Means of transmission" refers to the methods and technologies used to send data to a specified destination.

[0646] "Means of display" refers to methods and technologies for visually showing data on a device.

[0647] A "user" refers to an individual or organization that uses this system to access information.

[0648] This invention is a system that collects and analyzes explanatory texts and image data of exhibits in museums and exhibition facilities, and provides them in a user-accessible format. This system operates with the server, terminal, and user elements working in coordination with each other. Specific embodiments are described below.

[0649] Data collection methods

[0650] Server operation

[0651] The server calls Web APIs provided by museums and exhibition facilities to retrieve descriptive text and image data about the exhibits. Specifically, the server sends HTTP requests and receives responses in JSON format. The hardware used for this is a cloud service (e.g., AWS EC2). The software used is the Python language and an HTTP request library (e.g., requests).

[0652] Data analysis and text format

[0653] Server operation

[0654] The server analyzes the acquired image data and extracts text information from the image using optical character recognition (OCR) technology. This process utilizes APIs such as Google Cloud Vision. The server sends the image data to this API and organizes the returned text data.

[0655] Method of saving to the database

[0656] Server operation

[0657] The server stores the parsed text information and descriptions along with unique identifiers in a database. MySQL or PostgreSQL are used for this database. The server inserts the text data and exhibit IDs into the database tables.

[0658] User request processing methods

[0659] User actions

[0660] The user requests information about a specific exhibit using their device's browser. Specifically, they access a URL containing the ID of the specific exhibit. For example, they might access "http: / / example.com / exhibit?ID=AE001".

[0661] Server operation

[0662] The server receives a user request and searches for relevant data in the database. It executes an SQL query (e.g., "SELECT FROM exhibits WHERE id='AE001'") to retrieve the search results.

[0663] Data transmission and display methods

[0664] Server operation

[0665] The server sends the search results to the user's device in JSON format. The transmitted data includes descriptions of the exhibits, extracted text information, and associated images.

[0666] Terminal operation

[0667] The terminal analyzes the received data and generates HTML for display. The terminal displays the generated HTML in a browser, allowing the user to view detailed information about the exhibits.

[0668] Specific example

[0669] 1. Examples of data collection

[0670] The server retrieves information about the exhibits by calling "GET / api / exhibits". For example, information about the "Ancient Egypt Exhibition" is returned as a response. This response includes the exhibit ID, name, description, and image URL.

[0671] 2. Examples of data analysis and text conversion

[0672] The server performs OCR processing on the downloaded image, converting the text information "History of the Pharaohs" into text.

[0673] 3. Example of saving to a database

[0674] The textual information titled "History of the Pharaohs" will be stored in the database, associated with exhibit ID "AE001." This will allow for quick access during future searches.

[0675] 4. Example of processing user requests

[0676] The user accesses a URL containing the specific exhibit ID "AE001" from their device's browser and sends a request to the server.

[0677] 5. Examples of data transmission and display

[0678] The server searches the database for information related to "AE001" and sends it to the user's terminal. The terminal then displays the data, including descriptions and images, to the user.

[0679] Example of a prompt

[0680] Prompt: "Explain the important exhibits in the Ancient Egypt section, especially those related to the history of the pharaohs."

[0681] AI model response: "Information related to the history of the pharaohs shows how the kings of ancient Egypt ruled. Exhibit ID 'AE001' contains this information in detail."

[0682] The above describes a specific embodiment for carrying out this invention. This enables efficient and accurate collection, analysis, and storage of exhibit information, and allows users to access it quickly.

[0683] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0684] Step 1: Data Collection

[0685] Server operation

[0686] The server calls the Web API of the museum or exhibition facility to retrieve descriptive text and image data about the exhibits. First, the server sends an HTTP request to the API endpoint (e.g., "GET / api / exhibits"). The response is returned in JSON format. This JSON data includes the exhibit's ID, name, description, and image URL.

[0687] Input: API endpoint URL (e.g., "GET / api / exhibits")

[0688] Output: Exhibit data in JSON format

[0689] Specific actions:

[0690] The server sends an HTTP request: "GET / api / exhibits"

[0691] The server receives a response: data in JSON format.

[0692] The server extracts the exhibit's ID, name, description, and image URL from the response.

[0693] Step 2: Data Analysis and Text Conversion

[0694] Server operation

[0695] The server analyzes the acquired image data and extracts text information from the image using optical character recognition (OCR) technology. Using an OCR service such as the Google Cloud Vision API, the server sends the image and receives the text data.

[0696] Input: Downloaded image data

[0697] Output: Extracted text data

[0698] Specific actions:

[0699] The server sends image data to the Google Cloud Vision API.

[0700] The API recognizes text within an image and returns the text data.

[0701] The server organizes the returned text data into something like "The History of the Pharaohs."

[0702] Step 3: Saving to the database

[0703] Server operation

[0704] The server stores the parsed text information and explanatory text along with a unique identifier (ID) in a database. MySQL or PostgreSQL are used as the database system. Data is inserted via SQL queries.

[0705] Input: Parsed text data, unique identifier, description

[0706] Output: Records stored in the database

[0707] Specific actions:

[0708] The server inserts text data and exhibit IDs into the database using an SQL query.

[0709] Example: "INSERT INTO exhibits (id, name, description, text_data) VALUES ('AE001', 'History of the Pharaohs', 'Details about the Pharaohs', 'Text data in the image')"

[0710] Step 4: Processing user requests

[0711] User actions

[0712] The user requests information about a specific exhibit using their device's browser. Specifically, they access a URL that includes the exhibit ID (e.g., "http: / / example.com / exhibit?ID=AE001").

[0713] Server operation

[0714] The server parses the received request and searches for relevant data in the database. It then executes an SQL query to retrieve information about the requested exhibit.

[0715] Input: User request (URL)

[0716] Output: Exhibit data retrieved from the database

[0717] Specific actions:

[0718] The user accesses "http: / / example.com / exhibit?ID=AE001" in their browser.

[0719] The server receives the request and executes the SQL query "SELECT FROM exhibits WHERE id='AE001'".

[0720] The server retrieves the query results.

[0721] Step 5: Send and display data

[0722] Server operation

[0723] The server sends the search results to the user's device in JSON format. The transmitted data includes descriptions of the exhibits, extracted text information, and related images.

[0724] Terminal operation

[0725] The terminal analyzes the received data and generates HTML for display. The terminal then displays the generated HTML in a browser, allowing the user to view detailed information about the exhibits.

[0726] Input: Exhibit data (in JSON format) retrieved from the database.

[0727] Output: Exhibit information displayed visually to the user

[0728] Specific actions:

[0729] The server sends the search results to the terminal in JSON format.

[0730] The terminal parses the received JSON data and generates HTML.

[0731] The device displays the generated HTML in a browser, allowing the user to view detailed information about the exhibits.

[0732] The above outlines the specific processing steps of the system. This process allows users to efficiently access information about the exhibits.

[0733] (Application Example 1)

[0734] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0735] Currently, physical stores face the challenge of users not being able to easily obtain detailed product information and review data. In particular, small print on product labels or a lack of detailed information can reduce user purchasing intent. Furthermore, providing product information visually often requires additional effort from the store, making efficient information provision difficult. An efficient system is needed to solve these problems.

[0736] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0737] In this invention, the server includes means for acquiring data including descriptive text and image data; means for using optical character recognition technology to extract character information from the image data; means for storing the extracted character information and descriptive text in a database; means for searching the database for stored data based on a user request and transmitting the acquired data to the user; means for displaying the data transmitted to the user; and means for using a smartphone to acquire product identification information and display product descriptions. This allows users to easily obtain detailed information about products in physical stores and increase their purchasing intent. Furthermore, stores can provide product information efficiently, improving overall operational efficiency.

[0738] "Data including descriptive text" refers to text-based data that contains detailed information about exhibits or products.

[0739] "Image data" refers to data that includes photographs, diagrams, and other visual information of exhibits and products.

[0740] "Optical character recognition technology" is a technology that analyzes character information in an image and extracts it as text data.

[0741] A "database" is an information processing system for centrally storing and managing collected data.

[0742] A "unique identifier" is a unique ID number or code used to distinguish each piece of data from other pieces of data.

[0743] A "user request" is an access request made by a user seeking specific information.

[0744] "Searching" is the process of finding specific data from stored information.

[0745] "Sending" refers to the act of transferring the retrieved data to the user's device.

[0746] "Display" refers to the act of providing data visually to a user's device.

[0747] "Product identification information" refers to information such as barcodes and QR codes used to identify specific products.

[0748] A "smartphone" is a mobile phone-type information terminal used by users to obtain and display information.

[0749] To implement this invention, a system is required in which the server, terminal, and user elements work together in coordination. Below, a specific embodiment of this system, a product information provision system in a physical store, will be described.

[0750] Data collection

[0751] The server uses the store's API to retrieve product descriptions and image data. For example, it can retrieve data including product ID, product name, description, price, and image URL from the store's API. Furthermore, it can also collect product images in real time from cameras installed in the store.

[0752] Data analysis and text conversion

[0753] The server uses optical character recognition (OCR) technology to extract text information from collected product images. For example, it can retrieve information such as "organically grown" or "expiration date" from product packaging labels. The pytesseract library is used for this OCR processing.

[0754] Save to database

[0755] The server stores the extracted text information and descriptions in a database along with a unique identifier (product ID). This allows for quick data retrieval later. A database management system such as SQLite is used for the database.

[0756] Processing user requests

[0757] The user uses a smartphone app to scan product identification information, such as a QR code. The scanned information is sent to the server as a request. Based on this request, the server searches its database for the relevant product data.

[0758] Data transmission and display

[0759] The server sends the searched data to the user's smartphone. The transmitted data includes product names, descriptions, prices, images, ingredient information, and reviews. The user can visually confirm this information on their smartphone screen.

[0760] Specific example

[0761] Specific examples of data collection

[0762] The data retrieved from the store API is in the following format:

[0763] Product ID: "PRD001"

[0764] Product name: "Organic Green Tea"

[0765] Description: "100% organically grown, pesticide-free tea"

[0766] Price: 500 yen

[0767] Image URL: "https: / / store.com / images / greentea.jpg"

[0768] Specific examples of analysis using OCR

[0769] The server extracts the following text information from the product image:

[0770] "Organic cultivation"

[0771] "Best before date: December 2025"

[0772] Example of a prompt

[0773] Data collection:

[0774] response = requests.get('https: / / store-api.com / products')

[0775] OCR analysis:

[0776] text = pytesseract.image_to_string(Image.open('path / to / image.jpg'), lang='jpn')

[0777] In this way, the system according to this invention provides an environment in which users can easily obtain detailed information about products in physical stores, and enables stores to provide information efficiently.

[0778] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0779] Step 1:

[0780] Data collection

[0781] The server sends a request to the store's API to retrieve product descriptions and image data. Specifically, the server sends an HTTP GET request to the API endpoint and receives data in the response, including product ID, product name, description, price, and image URL. Input: API request, Output: Product data JSON.

[0782] Step 2:

[0783] Download image data

[0784] The server downloads product images based on the image URLs in the retrieved product data. Specifically, it sends an HTTP GET request to the image URL for each product and retrieves the image file as a response. Input: Image URL, Output: Image file.

[0785] Step 3:

[0786] Extraction of text information using OCR

[0787] The server extracts text information from downloaded image files using Optical Character Recognition (OCR) technology. Specifically, it uses the pytesseract library to extract text from image files. This text serves as supplementary information about the product. Input: Image file, Output: Extracted text.

[0788] Step 4:

[0789] Save to database

[0790] The server stores the extracted text information and retrieved product data in a database. Each data entry is assigned a unique identifier (product ID) for efficient management. Specifically, data is registered in an SQLite database using INSERT statements. Input: product data and extracted text; Output: database records.

[0791] Step 5:

[0792] User scan of product identification information

[0793] Users use a smartphone app to scan product identification information, such as QR codes, within the store. Specifically, they use their smartphone's camera to read QR codes, and the scanned data is displayed within the app. Input: QR code, Output: Product identification information.

[0794] Step 6:

[0795] Sending a user request

[0796] The user sends product identification information to the server as a request through the app. Specifically, the smartphone app sends an HTTP GET request containing the product ID to the server. Input: Product identification information, Output: Request sent.

[0797] Step 7:

[0798] Search from database

[0799] The server searches the database based on the product identification information received from the user and retrieves relevant product information. Specifically, it uses a SELECT statement to search the database using the product ID. Input: Product identification information, Output: Product data.

[0800] Step 8:

[0801] Sending product data to user terminals

[0802] The server sends the retrieved product data to the user's smartphone. Specifically, it sends the retrieved data as an HTTP response in JSON format. Input: Product data, Output: JSON response.

[0803] Step 9:

[0804] Displaying product information

[0805] The user visually displays product information received via a smartphone app. Specifically, the app displays product name, description, price, ingredient information, reviews, etc., on the screen. Input: JSON response, Output: Display of product information.

[0806] In this way, the server, terminal, and user work together at each step, making it possible for users to easily obtain product information in physical stores. This allows users to make purchasing decisions based on more detailed information, and stores can provide information more efficiently.

[0807] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0808] This invention combines a system that collects data including explanatory text and image data from museums and exhibition facilities, analyzes them, and provides them in a user-accessible format with an emotion engine that recognizes user emotions. This emotion engine makes it possible to dynamically optimize the information displayed according to the user's emotional state. Furthermore, it can learn from emotional data and provide information tailored to the user's preferences.

[0809] 1. Data Collection

[0810] The server retrieves descriptive text and image data about the exhibits using the museum's Web API and other data provision methods.

[0811] For example, the server retrieves a list of exhibits and their details by calling the museum's API, GET / api / exhibits.

[0812] 2. Acquisition of image data

[0813] The server uses the image URL in the response to send a GET / image1.jpg request and download the image data.

[0814] The image data file image1.jpg is retrieved and temporarily saved for later analysis.

[0815] 3. OCR processing

[0816] The server begins analyzing the acquired image data using optical character recognition (OCR) technology.

[0817] Extract text information from an image and save it as text data. Example: Extract the text "History of the Pharaohs" from the image image1.jpg.

[0818] 4. Saving to the database

[0819] The server stores the analyzed text information and explanatory data, along with a unique identifier, in a database.

[0820] Example: Save the data for "History of the Pharaohs" associated with ID "AE001" to the database in the following format:

[0821] SQL

[0822] INSERT INTO exhibits (id, name, description, ocr_text, image_url)

[0823] VALUES ('AE001', 'Ancient Egypt Exhibition', 'History of the Pharaohs', 'History of the Pharaohs', 'https: / / example.com / image1.jpg');

[0824] 5. Processing user requests

[0825] Users use their devices to request information about specific exhibits. For example, a request is sent to the server by accessing a specific URL through the device's browser.

[0826] The terminal sends a request to the server, and the server receives this request.

[0827] 6. Executing database queries

[0828] The server searches the database for information on the relevant exhibit based on the received request.

[0829] Example: Execute an SQL query to retrieve exhibit data for ID "AE001":

[0830] SQL

[0831] SELECT FROM exhibits WHERE id = 'AE001';

[0832] The acquired data is formatted and converted into a format that can be sent back to the user's terminal.

[0833] 7. Emotion recognition by an emotion engine

[0834] The server uses cameras and sensors installed on the user's device to recognize the user's emotional state in real time.

[0835] The emotion engine uses facial recognition and voice analysis technologies to extract emotions from the user's facial expressions and speech.

[0836] The recognized emotional state (e.g., joy, interest, surprise, confusion, etc.) is stored as data.

[0837] 8. Data transmission and display

[0838] The server dynamically optimizes the data it displays based on the emotional state recognized by the emotion engine.

[0839] The optimized data, including the exhibit descriptions, OCR text information, and related images, is sent to the user's device.

[0840] The terminal processes the received data to display it visually and then provides it to the user.

[0841] 9. Learning from user sentiment data

[0842] The server stores user emotion data recognized by the emotion engine and learns the user's preferences and interests.

[0843] The learned data will be used to provide information in the future. For example, if there is new information related to exhibits that the user has shown interest in in the past, the system will be adjusted to prioritize displaying that information.

[0844] Specific example

[0845] 1. Examples of data collection

[0846] The server calls the API GET / api / exhibits to retrieve information about the exhibits. For example, information about the "Ancient Egypt Exhibition" is returned as a response. This response includes the exhibit ID, name, description, and image URL.

[0847] The server uses the image URL to download images of the exhibits.

[0848] 2. Examples of data analysis and text conversion

[0849] The server performs OCR processing on the downloaded image. For example, it converts the text information "History of the Pharaohs" within the image into text.

[0850] 3. Examples of optimization using an emotion engine

[0851] When a user requests information about "AE001" and the device receives data from the server, the server analyzes the user's camera footage and recognizes that the user is in an "excited" state.

[0852] The server sends optimized data, including more detailed information and additional images, to users in an "excited" state.

[0853] 4. Example of data display

[0854] The device visually displays and provides optimized data to the user. For example, it might show a detailed description of "The History of the Pharaohs" along with several related images.

[0855] In this way, the system of the present invention can efficiently collect, analyze, and store detailed information about exhibits, making it readily accessible to users, and can also optimize the information according to the user's emotional state, providing a personalized experience.

[0856] The following describes the processing flow.

[0857] Step 1:

[0858] Data collection (server)

[0859] The server sends a GET request to the museum's Web API, named / api / exhibits, to retrieve a list of exhibits and detailed information about them.

[0860] The received response includes the exhibit ID, name, description, and image URL.

[0861] For example, retrieve {"id": "123", "name": "Ancient Egypt Exhibition", "description": "History of the Pharaohs", "image_url": "https: / / example.com / image1.jpg"}.

[0862] Step 2:

[0863] Image data acquisition (server)

[0864] The server uses the image URL in the response to send a GET / image1.jpg request to download the image data.

[0865] The image data file image1.jpg is retrieved and temporarily saved for later analysis.

[0866] Step 3:

[0867] OCR processing (server)

[0868] The server begins analyzing the acquired image data using optical character recognition (OCR) technology.

[0869] This process extracts text information from an image and saves it as text data. For example, it extracts the text "History of the Pharaohs" from the image image1.jpg.

[0870] Step 4:

[0871] Saving to the database (server)

[0872] The server stores the analyzed text information and explanatory data, along with a unique identifier, in a database.

[0873] As an example, the data for "History of the Pharaohs" associated with ID "AE001" will be saved to the database in the following format:

[0874] SQL

[0875] INSERT INTO exhibits (id, name, description, ocr_text, image_url)

[0876] VALUES ('AE001', 'Ancient Egypt Exhibition', 'History of the Pharaohs', 'History of the Pharaohs', 'https: / / example.com / image1.jpg');

[0877] Step 5:

[0878] Processing user requests (user, device, server)

[0879] The user uses their device to request information about a specific exhibit. For example, they might send a request like GET / api / exhibits / AE001 from their device's browser.

[0880] The server receives this request and prepares the corresponding data.

[0881] Step 6:

[0882] Execution of database queries (server)

[0883] Based on the request, the server searches the database for information about the relevant exhibit.

[0884] For example, execute an SQL query to retrieve exhibit data for ID "AE001":

[0885] SQL

[0886] SELECT FROM exhibits WHERE id = 'AE001';

[0887] The acquired data is formatted and converted into a format that can be sent back to the user's terminal.

[0888] Step 7:

[0889] Emotion recognition using an emotion engine (server, terminal)

[0890] The device uses cameras and sensors to capture the user's facial expressions and voice, and sends this information to the emotion engine.

[0891] The server uses an emotion engine to analyze and recognize the user's emotional state in real time.

[0892] For example, the emotion engine recognizes that a user is in a state of "excitement" after viewing the exhibit's description.

[0893] Step 8:

[0894] Data transmission and display (server, terminal, user)

[0895] The server dynamically optimizes the data it displays based on the recognized emotional state.

[0896] The optimized data, including the exhibit descriptions, OCR text information, and related images, is sent to the user's device.

[0897] The terminal visually displays the received data, allowing users to view detailed information about the exhibits.

[0898] Step 9:

[0899] Learning user sentiment data (server)

[0900] The server stores user emotion data recognized by the emotion engine and learns the user's preferences and interests.

[0901] For example, the system could be adjusted to prioritize displaying information related to exhibits the user has previously shown interest in during subsequent requests.

[0902] This process allows the system to provide information that responds to the user's real-time emotions, delivering a personalized experience.

[0903] (Example 2)

[0904] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0905] Traditional museum and exhibition information systems have faced challenges in maintaining user interest because the information provided is not dynamically optimized according to the user's emotional state or individual interests. Furthermore, the lack of consideration for user emotional states hinders the quality of understanding and experience of exhibits. Additionally, inefficient management and retrieval of acquired data, resulting in a mix of non-uniquely identifiable data, makes it difficult to quickly provide necessary information.

[0906] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0907] In this invention, the server includes means for acquiring data including descriptive text and image data; means for extracting character information from the acquired image data using optical character recognition technology; means for storing the extracted character information and descriptive text in a database; means for searching the database for stored data based on a user request and transmitting the acquired data to the user; means for recognizing the user's emotional state; means for dynamically optimizing the data to be displayed based on the recognized emotional state; and means for visually displaying the data on the user terminal. This enables personalized information provision according to the user's emotional state and interests, thereby improving the quality of the user experience. Furthermore, it enables efficient data management and retrieval, and allows for the rapid provision of uniquely identified data.

[0908] "Data including descriptive text" refers to data that contains descriptive information in text format about exhibits and related information.

[0909] "Image data" refers to data in image format that visually represents exhibits and related information.

[0910] "Optical character recognition technology" is a technology for extracting character information from image data, and is also known as OCR (Optical Character Recognition).

[0911] "Character information" refers to information in text format extracted using optical character recognition technology.

[0912] A "database" is a digital system for efficiently storing, managing, and retrieving acquired and analyzed data.

[0913] A "user request" is a request that a user sends to the system seeking specific information.

[0914] "User emotional state" refers to information about the user's current mental and emotional condition, and is recognized from facial expressions, voice analysis, and other sources.

[0915] "Dynamic optimization" means adjusting the information displayed in real time according to the user's current state and situation, and providing it in the most optimal form.

[0916] "Visually displaying" means displaying information graphically on the user's device, and it means providing images and text in a format that can be visually confirmed.

[0917] A "unique identifier" is a unique identifier assigned to each piece of data to distinguish it from other pieces of data.

[0918] This invention relates to a system for collecting, analyzing, and storing detailed information about exhibits in museums and exhibition facilities, and providing personalized information to users. This system acquires data including descriptive text and image data, analyzes them, and provides them in a user-accessible format. Furthermore, it can recognize user emotions and dynamically optimize the information. Specific embodiments are described below.

[0919] Hardware and software to be used

[0920] This system consists of three main elements: a server, terminals, and users. The server is a computer system with a high-performance processor and sufficient memory, which collects and analyzes data. It also uses an SQL-based database (e.g., PostgreSQL) as its database system. For image data analysis, it uses Tesseract, a widely used optical character recognition (OCR) technology. For emotion recognition, it uses an emotion analysis module called EmotionEngine. User terminals include smartphones, tablets, and personal computers equipped with cameras and sensors.

[0921] Data collection and analysis

[0922] The server calls the museum's API to retrieve descriptive text and image data about the exhibits. The retrieved image data is converted into text information using optical character recognition (OCR) technology. For example, it calls the API GET / api / exhibits, obtains the image URL from the response, accesses the corresponding URL to download the image, and extracts text information from that image. Through this process, the text data "History of the Pharaohs" is extracted from the image image1.jpg.

[0923] Data storage

[0924] The server stores the extracted text information and descriptions in the database, associating them with a unique identifier (e.g., ID "AE001"). For example, data related to the "Ancient Egypt Exhibition" is stored in the database in the following format:

[0925] Exhibit ID: AE001

[0926] Exhibition Title: Ancient Egypt Exhibition

[0927] Description: History of the Pharaohs

[0928] OCR text: History of the Pharaohs

[0929] Image URL: https: / / example.com / image1.jpg

[0930] Processing and optimizing user requests

[0931] Users request information about specific exhibits using their devices. For example, accessing a specific URL from a browser sends a request to the server. The server receives this request, searches its database for information about the relevant exhibit, and sends it back to the user. Furthermore, the system recognizes the user's emotional state in real time through the device's camera and sensors, and extracts it using an emotion engine. Based on the recognized emotional state (e.g., "excited"), the displayed data is dynamically optimized. For example, if the user is in an "excited" state, data containing more detailed information and additional images is sent to the user's device. The device then visually displays the optimized information based on this.

[0932] Learning user sentiment data

[0933] The server stores user emotion data recognized by the emotion engine and learns the user's preferences and interests. This learned data is used to provide information in the future. For example, if there is new information related to an exhibit that the user has shown interest in in the past, that information will be displayed preferentially.

[0934] In this way, the system of the present invention can collect, analyze, and store detailed information about exhibits, making it readily accessible to users, and can also optimize the information according to the user's emotional state to provide a personalized experience.

[0935] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0936] Step 1:

[0937] Data collection

[0938] The server calls the museum's Web API to retrieve descriptive text and image data about the exhibits.

[0939] Input: API call endpoint (e.g., GET / api / exhibits)

[0940] Data processing: Analyze the JSON response data obtained from the API and extract a list of exhibits.

[0941] Output: Exhibit list (e.g., exhibit ID, name, description, image URL, etc.)

[0942] Step 2:

[0943] Acquisition of image data

[0944] The server extracts image URLs from the list of exhibits and uses those URLs to download the image data.

[0945] Input: Image URL from the exhibit list (e.g., https: / / example.com / image1.jpg)

[0946] Data processing: Send an HTTP request and receive the corresponding image data.

[0947] Output: Image data file (e.g., image1.jpg)

[0948] Step 3:

[0949] OCR processing

[0950] The server applies optical character recognition (OCR) technology to the downloaded image data to extract text information.

[0951] Input: Image data file (e.g., image1.jpg)

[0952] Data processing: Analyze the text in the image using OCR software (e.g., Tesseract).

[0953] Output: Extracted text data (e.g., "History of the Pharaohs")

[0954] Step 4:

[0955] Save to database

[0956] The server stores the extracted text information and descriptions in a database along with a unique identifier.

[0957] Input: Exhibit ID, Name, Description, OCR text, Image URL

[0958] Data processing: Execute SQL queries to insert information into the database.

[0959] Output: Database entry where the data is stored (e.g., information associated with ID "AE001")

[0960] Step 5:

[0961] Processing user requests

[0962] Users use their devices to request information about specific exhibits.

[0963] Input: Request URL sent by the user from their browser (e.g., https: / / museum-website.com / exhibit?id=AE001)

[0964] Data processing: The terminal sends a request to the server, and the server receives the request.

[0965] Output: Request information forwarded to the server

[0966] Step 6:

[0967] Execution of database queries

[0968] Based on the received request, the server searches the database for information about the relevant exhibit.

[0969] Input: Exhibit ID in the request (e.g., AE001)

[0970] Data processing: Execute SQL queries to search for the relevant data.

[0971] Output: Searched exhibit information (e.g., exhibit name, description, OCR text, image URL)

[0972] Step 7:

[0973] Emotion recognition by an emotion engine

[0974] The server uses cameras and sensors installed on the user's device to recognize the user's emotional state in real time.

[0975] Input: User's camera video and audio data

[0976] Data processing: Analyze emotional information using an emotion engine (e.g., EmotionEngine).

[0977] Output: Recognized emotional state (e.g., "excited")

[0978] Step 8:

[0979] Data transmission and display

[0980] The server dynamically optimizes the data displayed based on the recognized emotional state.

[0981] Input: Exhibit information, perceived emotional state

[0982] Data processing: Generate optimized information and send it to the user's terminal.

[0983] Output: Dynamically optimized exhibit information

[0984] Step 9:

[0985] Displaying data

[0986] The terminal visually displays the received data and provides it to the user.

[0987] Input: Optimization information sent from the server

[0988] Data processing: Converting data into a layout and format for display.

[0989] Output: Detailed exhibit information provided to the user visually (e.g., detailed description, images)

[0990] Step 10:

[0991] Learning user sentiment data

[0992] The server stores user emotion data recognized by the emotion engine and learns the user's preferences and interests.

[0993] Input: Recognized emotional state, corresponding exhibit information

[0994] Data processing: Emotional data and exhibit information are stored in a database and analyzed.

[0995] Output: Recommendations based on learned user preferences

[0996] (Application Example 2)

[0997] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0998] Conventional exhibit information systems lacked dynamic data optimization based on user emotional states, resulting in insufficient improvement in the quality of the user experience. Furthermore, they lacked personalized information tailored to user preferences, making it difficult to quickly and accurately provide information of interest to users. Additionally, conventional systems lacked the functionality to understand and adjust displayed content based on real-time user emotional states, thus failing to provide information that met user interests and concerns.

[0999] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1000] In this invention, the server includes means for acquiring data including descriptive text and image data; means for analyzing the aforementioned data including descriptive text and image data and extracting text information from the image data; means for storing the extracted text information and descriptive text in a database; means for searching the database for stored data based on a user request and sending the acquired data to the user; means for displaying the data sent to the user; means for incorporating an emotion engine that analyzes sensor data acquired from a terminal to recognize the user's emotional state; and means for dynamically optimizing the information to be displayed according to the user's emotional state. This makes it possible to read the user's emotional state and provide information that corresponds to it in real time with dynamic optimization. Personalized information provision tailored to the user's interests and concerns can be realized, improving the quality of the user experience.

[1001] A "description" is text data that provides detailed information about an exhibit.

[1002] "Image data" refers to digital image files that include visual representations of exhibits and related items.

[1003] A "database" is an information system for systematically storing and managing descriptive text and image data.

[1004] A "user" or "device" refers to an individual requesting information about an exhibit and the device they use (e.g., a smartphone or tablet).

[1005] A "request" is a request from a user to retrieve information about a specific exhibit from a database.

[1006] An "emotion engine" is a system that recognizes the user's emotional state and dynamically optimizes the information displayed based on that data.

[1007] "Sensor data" refers to data such as the user's facial expressions and voice, acquired from the device's camera, microphone, etc.

[1008] "Dynamic optimization" is a process that adjusts the displayed information according to the user's real-time emotional state.

[1009] "Extraction" is the process of extracting text information from image data.

[1010] "Storage" refers to the process of recording and storing the analyzed data in a database.

[1011] This invention aims to improve the user experience in virtual stores. This system uses an emotion engine to recognize the user's emotional state and provides optimized product information tailored to that emotion. Specifically, the server implements this invention through the following main means.

[1012] The server first retrieves data including descriptions and image data. This data is collected from virtual stores via an API.

[1013] The server uses optical character recognition (OCR) technology to extract text information from the acquired image data. This allows for obtaining detailed text information about the product from the image data.

[1014] The parsed text information and explanatory text are stored in a database. The stored data is associated with a unique identifier, facilitating subsequent searching and retrieval.

[1015] The user requests specific product information using a device (e.g., a smartphone). In response to this request, the server searches the database, retrieves the relevant product data, and sends it to the user's device.

[1016] During this process, sensor data is acquired in real time from the camera and microphone installed in the device, and the emotion engine analyzes the user's emotional state. The emotion engine uses the Google Cloud Vision API and the Hugging Face Transformers library, among others, to recognize emotions from the user's facial expressions and voice.

[1017] The information displayed is dynamically optimized according to the recognized emotional state (e.g., "joy" or "surprise"). For example, if the emotion of "joy" is detected, the system will prioritize displaying new product and promotional information that the user is likely to be interested in.

[1018] To give a concrete example, when a user opens a virtual store app on their smartphone and points their face at the camera, the app's emotion engine recognizes "joy." As a result, the next shopping screen prioritizes displaying new products and promotional information that are of interest to the user.

[1019] Examples of prompt statements are as follows:

[1020] "By analyzing the user's facial expression, if an emotion of 'joy' is detected, we will suggest products that are suitable for that user."

[1021] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1022] Step 1:

[1023] The server retrieves data including descriptions and image data. It uses an API to collect product data from virtual stores. The input is the API endpoint URL, and the output is a list of product descriptions and image URLs.

[1024] Step 2:

[1025] The server extracts text information from the acquired image data using optical character recognition (OCR) technology. Specifically, it analyzes the image data using the Google Cloud Vision API and extracts the text information. The input is image data, and the output is the extracted text data.

[1026] Step 3:

[1027] The server stores the analyzed text information and explanatory text in a database. The database stores data using unique identifiers. Input consists of text information, explanatory text, and a unique identifier, while output is the verification result of the stored data.

[1028] Step 4:

[1029] The user requests specific product information using a terminal. The user's request is sent to the server. The input is the user's request content (such as the product ID), and the output is the result of sending the request to the server.

[1030] Step 5:

[1031] The server searches the database based on the user's request and retrieves the relevant product data. The input is the user's request, and the output is the relevant product data.

[1032] Step 6:

[1033] The server sends the acquired product data to the user's terminal. The input is the acquired product data, and the output is the result of the transmission to the user's terminal.

[1034] Step 7:

[1035] The device acquires sensor data in real time from its built-in camera and microphone, and transmits this data to the emotion engine. The input is the sensor data from the camera and microphone, and the output is the result of the data transmission to the emotion engine.

[1036] Step 8:

[1037] The emotion engine uses the Google Cloud Vision API and the Hugging Face Transformers library to analyze the user's emotional state from their facial expressions and voice. The input is sensor data, and the output is the recognized emotional state.

[1038] Step 9:

[1039] Based on the recognized emotional state, the server dynamically optimizes the information it displays. Specifically, it adjusts product information and suggestions according to the user's emotions. The input is emotional state data, and the output is optimized product information.

[1040] Step 10:

[1041] The server sends optimized product information to the user's terminal, which then displays it visually. The input is the optimized product information, and the output is the visual display result provided to the user.

[1042] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1043] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1044] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[1045] [Third Embodiment]

[1046] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[1047] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1048] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1049] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[1050] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1051] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1052] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1053] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1054] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1055] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1056] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1057] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[1058] This invention relates to a system that collects data, including explanatory text, and image data from museums and exhibition facilities, analyzes them, and provides them in a user-accessible format. This system operates with the server, terminal, and user elements working in coordination with each other as follows.

[1059] 1. Data Collection

[1060] The server retrieves descriptive text and image data about the exhibits using the museum's Web API and other data provision methods.

[1061] For example, the server retrieves a list of exhibits and their details by calling the museum's API, GET / api / exhibits.

[1062] 2. Data Analysis and Text Conversion

[1063] The server analyzes the acquired image data of the exhibits and extracts text information from the images using optical character recognition (OCR) technology.

[1064] For example, extract the textual information "History of the Pharaohs" from images of exhibits at the "Ancient Egypt Exhibition."

[1065] 3. Saving to the database

[1066] The server stores the analyzed text information and explanatory data, along with a unique identifier, in a database. This allows for a quick response to user search requests.

[1067] As an example, data on "History of the Pharaohs" associated with ID "AE001" will be saved to the database.

[1068] 4. Processing user requests

[1069] Users use their devices to request information about specific exhibits. For example, a request is sent to the server by accessing a specific URL through the device's browser.

[1070] The server receives this request and searches for the relevant data in the database.

[1071] 5. Data transmission and display

[1072] The server sends the searched data to the user's device. The transmitted data includes descriptions, text information, and related images.

[1073] The terminal visually displays the received data to the user. The user can then review the displayed information and understand the details of the exhibits.

[1074] Specific example

[1075] 1. Examples of data collection

[1076] The server calls the API GET / api / exhibits to retrieve information about the exhibits. For example, information about the "Ancient Egypt Exhibition" is returned as a response. This response includes the exhibit ID, name, description, and image URL.

[1077] The server uses the image URL to download images of the exhibits.

[1078] 2. Examples of data analysis and text conversion

[1079] The server performs OCR processing on the downloaded image. For example, it converts the text information "History of the Pharaohs" within the image into text.

[1080] 3. Example of saving to a database

[1081] The textual information titled "History of the Pharaohs" will be stored in the database, associated with exhibit ID "AE001." This will allow for quick access during future searches.

[1082] 4. Example of processing user requests

[1083] The user sends a request to the server by accessing a URL containing the specific exhibit ID "AE001" from their device's browser.

[1084] 5. Examples of data transmission and display

[1085] The server searches the database for information related to "AE001" and sends it to the user's terminal. The terminal then displays the data, including descriptions and images, to the user.

[1086] Thus, the system of the present invention efficiently collects, analyzes, and stores detailed information about exhibits, making it readily accessible to users, thereby ensuring that information about exhibits is provided fairly to people who have difficulty visiting in person.

[1087] The following describes the processing flow.

[1088] Step 1:

[1089] Data collection (server)

[1090] The server sends a GET request to the museum's Web API, named / api / exhibits, to retrieve a list of exhibits and detailed information about them.

[1091] The received response includes the exhibit ID, name, description, and image URL.

[1092] For example, retrieve {"id": "123", "name": "Ancient Egypt Exhibition", "description": "History of the Pharaohs", "image_url": "https: / / example.com / image1.jpg"}.

[1093] Step 2:

[1094] Image data acquisition (server)

[1095] The server uses the image URL in the response to send a GET / image1.jpg request to download the image data.

[1096] The image data file image1.jpg is retrieved and temporarily saved for later analysis.

[1097] Step 3:

[1098] OCR processing (server)

[1099] The server begins analyzing the acquired image data using optical character recognition (OCR) technology.

[1100] Extract text information from an image and save it as text data. Example: Extract the text "History of the Pharaohs" from the image image1.jpg.

[1101] Step 4:

[1102] Saving to the database (server)

[1103] The server stores the extracted text information and explanatory data, along with a unique identifier, in a database.

[1104] Example: Save the data for "History of the Pharaohs" associated with ID "AE001" to the database in the following format:

[1105] SQL

[1106] INSERT INTO exhibits (id, name, description, ocr_text, image_url)

[1107] VALUES ('AE001', 'Ancient Egypt Exhibition', 'History of the Pharaohs', 'History of the Pharaohs', 'https: / / example.com / image1.jpg');

[1108] Step 5:

[1109] Processing user requests (user, device, server)

[1110] The user uses a terminal to request information about a specific exhibit. For example, they might send a request to GET / api / exhibits / AE001.

[1111] The terminal sends a request to the server, and the server receives this request.

[1112] Step 6:

[1113] Execution of database queries (server)

[1114] The server searches the database for information on the relevant exhibit based on the received request.

[1115] Example: Execute an SQL query to retrieve exhibit data for ID "AE001":

[1116] SQL

[1117] SELECT FROM exhibits WHERE id = 'AE001';

[1118] The acquired data is formatted and converted into a format that can be sent back to the user's terminal.

[1119] Step 7:

[1120] Data transmission (server, terminal)

[1121] The server sends the formatted data to the user's terminal.

[1122] example:

[1123] json

[1124] {

[1125] "id": "AE001",

[1126] "name": "Ancient Egypt Exhibition",

[1127] "description": "History of the Pharaohs",

[1128] "ocr_text": "History of the Pharaohs",

[1129] "image_url": "https: / / example.com / image1.jpg"

[1130] }

[1131] Step 8:

[1132] Data display (terminal, user)

[1133] The terminal processes the received data to display it visually.

[1134] Users can view the explanatory text, OCR-generated text information, and related images displayed on the device screen.

[1135] This process allows users to efficiently and quickly obtain detailed information about exhibits without having to physically visit the museum.

[1136] (Example 1)

[1137] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1138] To provide timely and accurate information about exhibits in museums and exhibition facilities, a system is needed to effectively collect, analyze, and store explanatory texts and image data. However, conventional methods have low accuracy in analyzing collected data, making it difficult to respond quickly to user requests. Furthermore, there is a lack of means to fairly provide information about exhibits to people who have difficulty visiting in person. Therefore, the development of a system that enables efficient and highly accurate data collection, analysis, storage, and provision is necessary.

[1139] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1140] In this invention, the server includes means for acquiring data and image data, including descriptive text related to museums and exhibition facilities; means for extracting character information from image data using optical character recognition technology; means for storing the extracted character information and descriptive text in a database along with a unique identifier; means for searching the database for stored data based on a user request and transmitting the acquired data to the user; and means for displaying the data transmitted to the user. This enables the efficient collection, analysis, and storage of information on exhibits in museums and exhibition facilities, allowing users to access it quickly and providing information fairly to people who have difficulty visiting.

[1141] A museum is a facility for collecting, preserving, and displaying historical, cultural, and artistic exhibits.

[1142] An "exhibition facility" is a place where people can view exhibits for educational or entertainment purposes.

[1143] An "explanatory text" is a piece of writing that provides information and explanations about an exhibit.

[1144] "Data" refers to a collection of information or signals, and in this invention, it includes explanatory text and image data.

[1145] "Image data" refers to digital data that includes visual information such as photographs and illustrations of exhibits.

[1146] "Means of acquisition" refers to the methods and techniques used to collect the necessary data.

[1147] "Means of analysis" refers to methods and techniques for processing acquired data and extracting necessary information.

[1148] "Optical character recognition technology" is a technology that converts characters in an image into digital text.

[1149] "Textual information" refers to characters and sentences that are represented as digital text.

[1150] A "unique identifier" is a unique number or code used to independently identify data.

[1151] A "database" is a system for efficiently storing and managing large amounts of data, and for searching and extracting it as needed.

[1152] "Means of storage" refers to methods and technologies for recording data in storage devices such as databases.

[1153] A "request" refers to the operation or content of a request that a user makes to a system to request information.

[1154] "Means of searching" refers to methods and techniques for finding data within a database.

[1155] "Means of transmission" refers to the methods and technologies used to send data to a specified destination.

[1156] "Means of display" refers to methods and technologies for visually showing data on a device.

[1157] A "user" refers to an individual or organization that uses this system to access information.

[1158] This invention is a system that collects and analyzes explanatory texts and image data of exhibits in museums and exhibition facilities, and provides them in a user-accessible format. This system operates with the server, terminal, and user elements working in coordination with each other. Specific embodiments are described below.

[1159] Data collection methods

[1160] Server operation

[1161] The server calls Web APIs provided by museums and exhibition facilities to retrieve descriptive text and image data about the exhibits. Specifically, the server sends HTTP requests and receives responses in JSON format. The hardware used for this is a cloud service (e.g., AWS EC2). The software used is the Python language and an HTTP request library (e.g., requests).

[1162] Data analysis and text format

[1163] Server operation

[1164] The server analyzes the acquired image data and extracts text information from the image using optical character recognition (OCR) technology. This process utilizes APIs such as Google Cloud Vision. The server sends the image data to this API and organizes the returned text data.

[1165] Method of saving to the database

[1166] Server operation

[1167] The server stores the parsed text information and descriptions along with unique identifiers in a database. MySQL or PostgreSQL are used for this database. The server inserts the text data and exhibit IDs into the database tables.

[1168] User request processing methods

[1169] User actions

[1170] The user requests information about a specific exhibit using their device's browser. Specifically, they access a URL containing the ID of the specific exhibit. For example, they might access "http: / / example.com / exhibit?ID=AE001".

[1171] Server operation

[1172] The server receives a user request and searches for relevant data in the database. It executes an SQL query (e.g., "SELECT FROM exhibits WHERE id='AE001'") to retrieve the search results.

[1173] Data transmission and display methods

[1174] Server operation

[1175] The server sends the search results to the user's device in JSON format. The transmitted data includes descriptions of the exhibits, extracted text information, and associated images.

[1176] Terminal operation

[1177] The terminal analyzes the received data and generates HTML for display. The terminal displays the generated HTML in a browser, allowing the user to view detailed information about the exhibits.

[1178] Specific example

[1179] 1. Examples of data collection

[1180] The server retrieves information about the exhibits by calling "GET / api / exhibits". For example, information about the "Ancient Egypt Exhibition" is returned as a response. This response includes the exhibit ID, name, description, and image URL.

[1181] 2. Examples of data analysis and text conversion

[1182] The server performs OCR processing on the downloaded image, converting the text information "History of the Pharaohs" into text.

[1183] 3. Example of saving to a database

[1184] The textual information titled "History of the Pharaohs" will be stored in the database, associated with exhibit ID "AE001." This will allow for quick access during future searches.

[1185] 4. Example of processing user requests

[1186] The user accesses a URL containing the specific exhibit ID "AE001" from their device's browser and sends a request to the server.

[1187] 5. Examples of data transmission and display

[1188] The server searches the database for information related to "AE001" and sends it to the user's terminal. The terminal then displays the data, including descriptions and images, to the user.

[1189] Example of a prompt

[1190] Prompt: "Explain the important exhibits in the Ancient Egypt section, especially those related to the history of the pharaohs."

[1191] AI model response: "Information related to the history of the pharaohs shows how the kings of ancient Egypt ruled. Exhibit ID 'AE001' contains this information in detail."

[1192] The above describes a specific embodiment for carrying out this invention. This enables efficient and accurate collection, analysis, and storage of exhibit information, and allows users to access it quickly.

[1193] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1194] Step 1: Data Collection

[1195] Server operation

[1196] The server calls the Web API of the museum or exhibition facility to retrieve descriptive text and image data about the exhibits. First, the server sends an HTTP request to the API endpoint (e.g., "GET / api / exhibits"). The response is returned in JSON format. This JSON data includes the exhibit's ID, name, description, and image URL.

[1197] Input: API endpoint URL (e.g., "GET / api / exhibits")

[1198] Output: Exhibit data in JSON format

[1199] Specific actions:

[1200] The server sends an HTTP request: "GET / api / exhibits"

[1201] The server receives a response: data in JSON format.

[1202] The server extracts the exhibit's ID, name, description, and image URL from the response.

[1203] Step 2: Data Analysis and Text Conversion

[1204] Server operation

[1205] The server analyzes the acquired image data and extracts text information from the image using optical character recognition (OCR) technology. Using an OCR service such as the Google Cloud Vision API, the server sends the image and receives the text data.

[1206] Input: Downloaded image data

[1207] Output: Extracted text data

[1208] Specific actions:

[1209] The server sends image data to the Google Cloud Vision API.

[1210] The API recognizes text within an image and returns the text data.

[1211] The server organizes the returned text data into something like "The History of the Pharaohs."

[1212] Step 3: Saving to the database

[1213] Server operation

[1214] The server stores the parsed text information and explanatory text along with a unique identifier (ID) in a database. MySQL or PostgreSQL are used as the database system. Data is inserted via SQL queries.

[1215] Input: Parsed text data, unique identifier, description

[1216] Output: Records stored in the database

[1217] Specific actions:

[1218] The server inserts text data and exhibit IDs into the database using an SQL query.

[1219] Example: "INSERT INTO exhibits (id, name, description, text_data) VALUES ('AE001', 'History of the Pharaohs', 'Details about the Pharaohs', 'Text data in the image')"

[1220] Step 4: Processing user requests

[1221] User actions

[1222] The user requests information about a specific exhibit using their device's browser. Specifically, they access a URL that includes the exhibit ID (e.g., "http: / / example.com / exhibit?ID=AE001").

[1223] Server operation

[1224] The server parses the received request and searches for relevant data in the database. It then executes an SQL query to retrieve information about the requested exhibit.

[1225] Input: User request (URL)

[1226] Output: Exhibit data retrieved from the database

[1227] Specific actions:

[1228] The user accesses "http: / / example.com / exhibit?ID=AE001" in their browser.

[1229] The server receives the request and executes the SQL query "SELECT FROM exhibits WHERE id='AE001'".

[1230] The server retrieves the query results.

[1231] Step 5: Send and display data

[1232] Server operation

[1233] The server sends the search results to the user's device in JSON format. The transmitted data includes descriptions of the exhibits, extracted text information, and related images.

[1234] Terminal operation

[1235] The terminal analyzes the received data and generates HTML for display. The terminal then displays the generated HTML in a browser, allowing the user to view detailed information about the exhibits.

[1236] Input: Exhibit data (in JSON format) retrieved from the database.

[1237] Output: Exhibit information displayed visually to the user

[1238] Specific actions:

[1239] The server sends the search results to the terminal in JSON format.

[1240] The terminal parses the received JSON data and generates HTML.

[1241] The device displays the generated HTML in a browser, allowing the user to view detailed information about the exhibits.

[1242] The above outlines the specific processing steps of the system. This process allows users to efficiently access information about the exhibits.

[1243] (Application Example 1)

[1244] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1245] Currently, physical stores face the challenge of users not being able to easily obtain detailed product information and review data. In particular, small print on product labels or a lack of detailed information can reduce user purchasing intent. Furthermore, providing product information visually often requires additional effort from the store, making efficient information provision difficult. An efficient system is needed to solve these problems.

[1246] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1247] In this invention, the server includes means for acquiring data including descriptive text and image data; means for using optical character recognition technology to extract character information from the image data; means for storing the extracted character information and descriptive text in a database; means for searching the database for stored data based on a user request and transmitting the acquired data to the user; means for displaying the data transmitted to the user; and means for using a smartphone to acquire product identification information and display product descriptions. This allows users to easily obtain detailed information about products in physical stores and increase their purchasing intent. Furthermore, stores can provide product information efficiently, improving overall operational efficiency.

[1248] "Data including descriptive text" refers to text-based data that contains detailed information about exhibits or products.

[1249] "Image data" refers to data that includes photographs, diagrams, and other visual information of exhibits and products.

[1250] "Optical character recognition technology" is a technology that analyzes character information in an image and extracts it as text data.

[1251] A "database" is an information processing system for centrally storing and managing collected data.

[1252] A "unique identifier" is a unique ID number or code used to distinguish each piece of data from other pieces of data.

[1253] A "user request" is an access request made by a user seeking specific information.

[1254] "Searching" is the process of finding specific data from stored information.

[1255] "Sending" refers to the act of transferring the retrieved data to the user's device.

[1256] "Display" refers to the act of providing data visually to a user's device.

[1257] "Product identification information" refers to information such as barcodes and QR codes used to identify specific products.

[1258] A "smartphone" is a mobile phone-type information terminal used by users to obtain and display information.

[1259] To implement this invention, a system is required in which the server, terminal, and user elements work together in coordination. Below, a specific embodiment of this system, a product information provision system in a physical store, will be described.

[1260] Data collection

[1261] The server uses the store's API to retrieve product descriptions and image data. For example, it can retrieve data including product ID, product name, description, price, and image URL from the store's API. Furthermore, it can also collect product images in real time from cameras installed in the store.

[1262] Data analysis and text conversion

[1263] The server uses optical character recognition (OCR) technology to extract text information from collected product images. For example, it can retrieve information such as "organically grown" or "expiration date" from product packaging labels. The pytesseract library is used for this OCR processing.

[1264] Save to database

[1265] The server stores the extracted text information and descriptions in a database along with a unique identifier (product ID). This allows for quick data retrieval later. A database management system such as SQLite is used for the database.

[1266] Processing user requests

[1267] The user uses a smartphone app to scan product identification information, such as a QR code. The scanned information is sent to the server as a request. Based on this request, the server searches its database for the relevant product data.

[1268] Data transmission and display

[1269] The server sends the searched data to the user's smartphone. The transmitted data includes product names, descriptions, prices, images, ingredient information, and reviews. The user can visually confirm this information on their smartphone screen.

[1270] Specific example

[1271] Specific examples of data collection

[1272] The data retrieved from the store API is in the following format:

[1273] Product ID: "PRD001"

[1274] Product name: "Organic Green Tea"

[1275] Description: "100% organically grown, pesticide-free tea"

[1276] Price: 500 yen

[1277] Image URL: "https: / / store.com / images / greentea.jpg"

[1278] Specific examples of analysis using OCR

[1279] The server extracts the following text information from the product image:

[1280] "Organic cultivation"

[1281] "Best before date: December 2025"

[1282] Example of a prompt

[1283] Data collection:

[1284] response = requests.get('https: / / store-api.com / products')

[1285] OCR analysis:

[1286] text = pytesseract.image_to_string(Image.open('path / to / image.jpg'), lang='jpn')

[1287] In this way, the system according to this invention provides an environment in which users can easily obtain detailed information about products in physical stores, and enables stores to provide information efficiently.

[1288] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1289] Step 1:

[1290] Data collection

[1291] The server sends a request to the store's API to retrieve product descriptions and image data. Specifically, the server sends an HTTP GET request to the API endpoint and receives data in the response, including product ID, product name, description, price, and image URL. Input: API request, Output: Product data JSON.

[1292] Step 2:

[1293] Download image data

[1294] The server downloads product images based on the image URLs in the retrieved product data. Specifically, it sends an HTTP GET request to the image URL for each product and retrieves the image file as a response. Input: Image URL, Output: Image file.

[1295] Step 3:

[1296] Extraction of text information using OCR

[1297] The server extracts text information from downloaded image files using Optical Character Recognition (OCR) technology. Specifically, it uses the pytesseract library to extract text from image files. This text serves as supplementary information about the product. Input: Image file, Output: Extracted text.

[1298] Step 4:

[1299] Save to database

[1300] The server stores the extracted text information and retrieved product data in a database. Each data entry is assigned a unique identifier (product ID) for efficient management. Specifically, data is registered in an SQLite database using INSERT statements. Input: product data and extracted text; Output: database records.

[1301] Step 5:

[1302] User scan of product identification information

[1303] Users use a smartphone app to scan product identification information, such as QR codes, within the store. Specifically, they use their smartphone's camera to read QR codes, and the scanned data is displayed within the app. Input: QR code, Output: Product identification information.

[1304] Step 6:

[1305] Sending a user request

[1306] The user sends product identification information to the server as a request through the app. Specifically, the smartphone app sends an HTTP GET request containing the product ID to the server. Input: Product identification information, Output: Request sent.

[1307] Step 7:

[1308] Search from database

[1309] The server searches the database based on the product identification information received from the user and retrieves relevant product information. Specifically, it uses a SELECT statement to search the database using the product ID. Input: Product identification information, Output: Product data.

[1310] Step 8:

[1311] Sending product data to user terminals

[1312] The server sends the retrieved product data to the user's smartphone. Specifically, it sends the retrieved data as an HTTP response in JSON format. Input: Product data, Output: JSON response.

[1313] Step 9:

[1314] Displaying product information

[1315] The user visually displays product information received via a smartphone app. Specifically, the app displays product name, description, price, ingredient information, reviews, etc., on the screen. Input: JSON response, Output: Display of product information.

[1316] In this way, the server, terminal, and user work together at each step, making it possible for users to easily obtain product information in physical stores. This allows users to make purchasing decisions based on more detailed information, and stores can provide information more efficiently.

[1317] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1318] This invention combines a system that collects data including explanatory text and image data from museums and exhibition facilities, analyzes them, and provides them in a user-accessible format with an emotion engine that recognizes user emotions. This emotion engine makes it possible to dynamically optimize the information displayed according to the user's emotional state. Furthermore, it can learn from emotional data and provide information tailored to the user's preferences.

[1319] 1. Data Collection

[1320] The server retrieves descriptive text and image data about the exhibits using the museum's Web API and other data provision methods.

[1321] For example, the server retrieves a list of exhibits and their details by calling the museum's API, GET / api / exhibits.

[1322] 2. Acquisition of image data

[1323] The server uses the image URL in the response to send a GET / image1.jpg request and download the image data.

[1324] The image data file image1.jpg is retrieved and temporarily saved for later analysis.

[1325] 3. OCR processing

[1326] The server begins analyzing the acquired image data using optical character recognition (OCR) technology.

[1327] Extract text information from an image and save it as text data. Example: Extract the text "History of the Pharaohs" from the image image1.jpg.

[1328] 4. Saving to the database

[1329] The server stores the analyzed text information and explanatory data, along with a unique identifier, in a database.

[1330] Example: Save the data for "History of the Pharaohs" associated with ID "AE001" to the database in the following format:

[1331] SQL

[1332] INSERT INTO exhibits (id, name, description, ocr_text, image_url)

[1333] VALUES ('AE001', 'Ancient Egypt Exhibition', 'History of the Pharaohs', 'History of the Pharaohs', 'https: / / example.com / image1.jpg');

[1334] 5. Processing user requests

[1335] Users use their devices to request information about specific exhibits. For example, a request is sent to the server by accessing a specific URL through the device's browser.

[1336] The terminal sends a request to the server, and the server receives this request.

[1337] 6. Executing database queries

[1338] The server searches the database for information on the relevant exhibit based on the received request.

[1339] Example: Execute an SQL query to retrieve exhibit data for ID "AE001":

[1340] SQL

[1341] SELECT FROM exhibits WHERE id = 'AE001';

[1342] The acquired data is formatted and converted into a format that can be sent back to the user's terminal.

[1343] 7. Emotion recognition by an emotion engine

[1344] The server uses cameras and sensors installed on the user's device to recognize the user's emotional state in real time.

[1345] The emotion engine uses facial recognition and voice analysis technologies to extract emotions from the user's facial expressions and speech.

[1346] The recognized emotional state (e.g., joy, interest, surprise, confusion, etc.) is stored as data.

[1347] 8. Data transmission and display

[1348] The server dynamically optimizes the data it displays based on the emotional state recognized by the emotion engine.

[1349] The optimized data, including the exhibit descriptions, OCR text information, and related images, is sent to the user's device.

[1350] The terminal processes the received data to display it visually and then provides it to the user.

[1351] 9. Learning from user sentiment data

[1352] The server stores user emotion data recognized by the emotion engine and learns the user's preferences and interests.

[1353] The learned data will be used to provide information in the future. For example, if there is new information related to exhibits that the user has shown interest in in the past, the system will be adjusted to prioritize displaying that information.

[1354] Specific example

[1355] 1. Examples of data collection

[1356] The server calls the API GET / api / exhibits to retrieve information about the exhibits. For example, information about the "Ancient Egypt Exhibition" is returned as a response. This response includes the exhibit ID, name, description, and image URL.

[1357] The server uses the image URL to download images of the exhibits.

[1358] 2. Examples of data analysis and text conversion

[1359] The server performs OCR processing on the downloaded image. For example, it converts the text information "History of the Pharaohs" within the image into text.

[1360] 3. Examples of optimization using an emotion engine

[1361] When a user requests information about "AE001" and the device receives data from the server, the server analyzes the user's camera footage and recognizes that the user is in an "excited" state.

[1362] The server sends optimized data, including more detailed information and additional images, to users in an "excited" state.

[1363] 4. Example of data display

[1364] The device visually displays and provides optimized data to the user. For example, it might show a detailed description of "The History of the Pharaohs" along with several related images.

[1365] In this way, the system of the present invention can efficiently collect, analyze, and store detailed information about exhibits, making it readily accessible to users, and can also optimize the information according to the user's emotional state, providing a personalized experience.

[1366] The following describes the processing flow.

[1367] Step 1:

[1368] Data collection (server)

[1369] The server sends a GET request to the museum's Web API, named / api / exhibits, to retrieve a list of exhibits and detailed information about them.

[1370] The received response includes the exhibit ID, name, description, and image URL.

[1371] For example, retrieve {"id": "123", "name": "Ancient Egypt Exhibition", "description": "History of the Pharaohs", "image_url": "https: / / example.com / image1.jpg"}.

[1372] Step 2:

[1373] Image data acquisition (server)

[1374] The server uses the image URL in the response to send a GET / image1.jpg request to download the image data.

[1375] The image data file image1.jpg is retrieved and temporarily saved for later analysis.

[1376] Step 3:

[1377] OCR processing (server)

[1378] The server begins analyzing the acquired image data using optical character recognition (OCR) technology.

[1379] This process extracts text information from an image and saves it as text data. For example, it extracts the text "History of the Pharaohs" from the image image1.jpg.

[1380] Step 4:

[1381] Saving to the database (server)

[1382] The server stores the analyzed text information and explanatory data, along with a unique identifier, in a database.

[1383] As an example, the data for "History of the Pharaohs" associated with ID "AE001" will be saved to the database in the following format:

[1384] SQL

[1385] INSERT INTO exhibits (id, name, description, ocr_text, image_url)

[1386] VALUES ('AE001', 'Ancient Egypt Exhibition', 'History of the Pharaohs', 'History of the Pharaohs', 'https: / / example.com / image1.jpg');

[1387] Step 5:

[1388] Processing user requests (user, device, server)

[1389] The user uses their device to request information about a specific exhibit. For example, they might send a request like GET / api / exhibits / AE001 from their device's browser.

[1390] The server receives this request and prepares the corresponding data.

[1391] Step 6:

[1392] Execution of database queries (server)

[1393] Based on the request, the server searches the database for information about the relevant exhibit.

[1394] For example, execute an SQL query to retrieve exhibit data for ID "AE001":

[1395] SQL

[1396] SELECT FROM exhibits WHERE id = 'AE001';

[1397] The acquired data is formatted and converted into a format that can be sent back to the user's terminal.

[1398] Step 7:

[1399] Emotion recognition using an emotion engine (server, terminal)

[1400] The device uses cameras and sensors to capture the user's facial expressions and voice, and sends this information to the emotion engine.

[1401] The server uses an emotion engine to analyze and recognize the user's emotional state in real time.

[1402] For example, the emotion engine recognizes that a user is in a state of "excitement" after viewing the exhibit's description.

[1403] Step 8:

[1404] Data transmission and display (server, terminal, user)

[1405] The server dynamically optimizes the data it displays based on the recognized emotional state.

[1406] The optimized data, including the exhibit descriptions, OCR text information, and related images, is sent to the user's device.

[1407] The terminal visually displays the received data, allowing users to view detailed information about the exhibits.

[1408] Step 9:

[1409] Learning user sentiment data (server)

[1410] The server stores user emotion data recognized by the emotion engine and learns the user's preferences and interests.

[1411] For example, the system could be adjusted to prioritize displaying information related to exhibits the user has previously shown interest in during subsequent requests.

[1412] This process allows the system to provide information that responds to the user's real-time emotions, delivering a personalized experience.

[1413] (Example 2)

[1414] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1415] Traditional museum and exhibition information systems have faced challenges in maintaining user interest because the information provided is not dynamically optimized according to the user's emotional state or individual interests. Furthermore, the lack of consideration for user emotional states hinders the quality of understanding and experience of exhibits. Additionally, inefficient management and retrieval of acquired data, resulting in a mix of non-uniquely identifiable data, makes it difficult to quickly provide necessary information.

[1416] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1417] In this invention, the server includes means for acquiring data including descriptive text and image data; means for extracting character information from the acquired image data using optical character recognition technology; means for storing the extracted character information and descriptive text in a database; means for searching the database for stored data based on a user request and transmitting the acquired data to the user; means for recognizing the user's emotional state; means for dynamically optimizing the data to be displayed based on the recognized emotional state; and means for visually displaying the data on the user terminal. This enables personalized information provision according to the user's emotional state and interests, thereby improving the quality of the user experience. Furthermore, it enables efficient data management and retrieval, and allows for the rapid provision of uniquely identified data.

[1418] "Data including descriptive text" refers to data that contains descriptive information in text format about exhibits and related information.

[1419] "Image data" refers to data in image format that visually represents exhibits and related information.

[1420] "Optical character recognition technology" is a technology for extracting character information from image data, and is also known as OCR (Optical Character Recognition).

[1421] "Character information" refers to information in text format extracted using optical character recognition technology.

[1422] A "database" is a digital system for efficiently storing, managing, and retrieving acquired and analyzed data.

[1423] A "user request" is a request that a user sends to the system seeking specific information.

[1424] "User emotional state" refers to information about the user's current mental and emotional condition, and is recognized from facial expressions, voice analysis, and other sources.

[1425] "Dynamic optimization" means adjusting the information displayed in real time according to the user's current state and situation, and providing it in the most optimal form.

[1426] "Visually displaying" means displaying information graphically on the user's device, and it means providing images and text in a format that can be visually confirmed.

[1427] A "unique identifier" is a unique identifier assigned to each piece of data to distinguish it from other pieces of data.

[1428] This invention relates to a system for collecting, analyzing, and storing detailed information about exhibits in museums and exhibition facilities, and providing personalized information to users. This system acquires data including descriptive text and image data, analyzes them, and provides them in a user-accessible format. Furthermore, it can recognize user emotions and dynamically optimize the information. Specific embodiments are described below.

[1429] Hardware and software to be used

[1430] This system consists of three main elements: a server, terminals, and users. The server is a computer system with a high-performance processor and sufficient memory, which collects and analyzes data. It also uses an SQL-based database (e.g., PostgreSQL) as its database system. For image data analysis, it uses Tesseract, a widely used optical character recognition (OCR) technology. For emotion recognition, it uses an emotion analysis module called EmotionEngine. User terminals include smartphones, tablets, and personal computers equipped with cameras and sensors.

[1431] Data collection and analysis

[1432] The server calls the museum's API to retrieve descriptive text and image data about the exhibits. The retrieved image data is converted into text information using optical character recognition (OCR) technology. For example, it calls the API GET / api / exhibits, obtains the image URL from the response, accesses the corresponding URL to download the image, and extracts text information from that image. Through this process, the text data "History of the Pharaohs" is extracted from the image image1.jpg.

[1433] Data storage

[1434] The server stores the extracted text information and descriptions in the database, associating them with a unique identifier (e.g., ID "AE001"). For example, data related to the "Ancient Egypt Exhibition" is stored in the database in the following format:

[1435] Exhibit ID: AE001

[1436] Exhibition Title: Ancient Egypt Exhibition

[1437] Description: History of the Pharaohs

[1438] OCR text: History of the Pharaohs

[1439] Image URL: https: / / example.com / image1.jpg

[1440] Processing and optimizing user requests

[1441] Users request information about specific exhibits using their devices. For example, accessing a specific URL from a browser sends a request to the server. The server receives this request, searches its database for information about the relevant exhibit, and sends it back to the user. Furthermore, the system recognizes the user's emotional state in real time through the device's camera and sensors, and extracts it using an emotion engine. Based on the recognized emotional state (e.g., "excited"), the displayed data is dynamically optimized. For example, if the user is in an "excited" state, data containing more detailed information and additional images is sent to the user's device. The device then visually displays the optimized information based on this.

[1442] Learning user sentiment data

[1443] The server stores user emotion data recognized by the emotion engine and learns the user's preferences and interests. This learned data is used to provide information in the future. For example, if there is new information related to an exhibit that the user has shown interest in in the past, that information will be displayed preferentially.

[1444] In this way, the system of the present invention can collect, analyze, and store detailed information about exhibits, making it readily accessible to users, and can also optimize the information according to the user's emotional state to provide a personalized experience.

[1445] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1446] Step 1:

[1447] Data collection

[1448] The server calls the museum's Web API to retrieve descriptive text and image data about the exhibits.

[1449] Input: API call endpoint (e.g., GET / api / exhibits)

[1450] Data processing: Analyze the JSON response data obtained from the API and extract a list of exhibits.

[1451] Output: Exhibit list (e.g., exhibit ID, name, description, image URL, etc.)

[1452] Step 2:

[1453] Acquisition of image data

[1454] The server extracts image URLs from the list of exhibits and uses those URLs to download the image data.

[1455] Input: Image URL from the exhibit list (e.g., https: / / example.com / image1.jpg)

[1456] Data processing: Send an HTTP request and receive the corresponding image data.

[1457] Output: Image data file (e.g., image1.jpg)

[1458] Step 3:

[1459] OCR processing

[1460] The server applies optical character recognition (OCR) technology to the downloaded image data to extract text information.

[1461] Input: Image data file (e.g., image1.jpg)

[1462] Data processing: Analyze the text in the image using OCR software (e.g., Tesseract).

[1463] Output: Extracted text data (e.g., "History of the Pharaohs")

[1464] Step 4:

[1465] Save to database

[1466] The server stores the extracted text information and descriptions in a database along with a unique identifier.

[1467] Input: Exhibit ID, Name, Description, OCR text, Image URL

[1468] Data processing: Execute SQL queries to insert information into the database.

[1469] Output: Database entry where the data is stored (e.g., information associated with ID "AE001")

[1470] Step 5:

[1471] Processing user requests

[1472] Users use their devices to request information about specific exhibits.

[1473] Input: Request URL sent by the user from their browser (e.g., https: / / museum-website.com / exhibit?id=AE001)

[1474] Data processing: The terminal sends a request to the server, and the server receives the request.

[1475] Output: Request information forwarded to the server

[1476] Step 6:

[1477] Execution of database queries

[1478] Based on the received request, the server searches the database for information about the relevant exhibit.

[1479] Input: Exhibit ID in the request (e.g., AE001)

[1480] Data processing: Execute SQL queries to search for the relevant data.

[1481] Output: Searched exhibit information (e.g., exhibit name, description, OCR text, image URL)

[1482] Step 7:

[1483] Emotion recognition by an emotion engine

[1484] The server uses cameras and sensors installed on the user's device to recognize the user's emotional state in real time.

[1485] Input: User's camera video and audio data

[1486] Data processing: Analyze emotional information using an emotion engine (e.g., EmotionEngine).

[1487] Output: Recognized emotional state (e.g., "excited")

[1488] Step 8:

[1489] Data transmission and display

[1490] The server dynamically optimizes the data displayed based on the recognized emotional state.

[1491] Input: Exhibit information, perceived emotional state

[1492] Data processing: Generate optimized information and send it to the user's terminal.

[1493] Output: Dynamically optimized exhibit information

[1494] Step 9:

[1495] Displaying data

[1496] The terminal visually displays the received data and provides it to the user.

[1497] Input: Optimization information sent from the server

[1498] Data processing: Converting data into a layout and format for display.

[1499] Output: Detailed exhibit information provided to the user visually (e.g., detailed description, images)

[1500] Step 10:

[1501] Learning user sentiment data

[1502] The server stores user emotion data recognized by the emotion engine and learns the user's preferences and interests.

[1503] Input: Recognized emotional state, corresponding exhibit information

[1504] Data processing: Emotional data and exhibit information are stored in a database and analyzed.

[1505] Output: Recommendations based on learned user preferences

[1506] (Application Example 2)

[1507] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1508] Conventional exhibit information systems lacked dynamic data optimization based on user emotional states, resulting in insufficient improvement in the quality of the user experience. Furthermore, they lacked personalized information tailored to user preferences, making it difficult to quickly and accurately provide information of interest to users. Additionally, conventional systems lacked the functionality to understand and adjust displayed content based on real-time user emotional states, thus failing to provide information that met user interests and concerns.

[1509] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1510] In this invention, the server includes means for acquiring data including descriptive text and image data; means for analyzing the aforementioned data including descriptive text and image data and extracting text information from the image data; means for storing the extracted text information and descriptive text in a database; means for searching the database for stored data based on a user request and sending the acquired data to the user; means for displaying the data sent to the user; means for incorporating an emotion engine that analyzes sensor data acquired from a terminal to recognize the user's emotional state; and means for dynamically optimizing the information to be displayed according to the user's emotional state. This makes it possible to read the user's emotional state and provide information that corresponds to it in real time with dynamic optimization. Personalized information provision tailored to the user's interests and concerns can be realized, improving the quality of the user experience.

[1511] A "description" is text data that provides detailed information about an exhibit.

[1512] "Image data" refers to digital image files that include visual representations of exhibits and related items.

[1513] A "database" is an information system for systematically storing and managing descriptive text and image data.

[1514] A "user" or "device" refers to an individual requesting information about an exhibit and the device they use (e.g., a smartphone or tablet).

[1515] A "request" is a request from a user to retrieve information about a specific exhibit from a database.

[1516] An "emotion engine" is a system that recognizes the user's emotional state and dynamically optimizes the information displayed based on that data.

[1517] "Sensor data" refers to data such as the user's facial expressions and voice, acquired from the device's camera, microphone, etc.

[1518] "Dynamic optimization" is a process that adjusts the displayed information according to the user's real-time emotional state.

[1519] "Extraction" is the process of extracting text information from image data.

[1520] "Storage" refers to the process of recording and storing the analyzed data in a database.

[1521] This invention aims to improve the user experience in virtual stores. This system uses an emotion engine to recognize the user's emotional state and provides optimized product information tailored to that emotion. Specifically, the server implements this invention through the following main means.

[1522] The server first retrieves data including descriptions and image data. This data is collected from virtual stores via an API.

[1523] The server uses optical character recognition (OCR) technology to extract text information from the acquired image data. This allows for obtaining detailed text information about the product from the image data.

[1524] The parsed text information and explanatory text are stored in a database. The stored data is associated with a unique identifier, facilitating subsequent searching and retrieval.

[1525] The user requests specific product information using a device (e.g., a smartphone). In response to this request, the server searches the database, retrieves the relevant product data, and sends it to the user's device.

[1526] During this process, sensor data is acquired in real time from the camera and microphone installed in the device, and the emotion engine analyzes the user's emotional state. The emotion engine uses the Google Cloud Vision API and the Hugging Face Transformers library, among others, to recognize emotions from the user's facial expressions and voice.

[1527] The information displayed is dynamically optimized according to the recognized emotional state (e.g., "joy" or "surprise"). For example, if the emotion of "joy" is detected, the system will prioritize displaying new product and promotional information that the user is likely to be interested in.

[1528] To give a concrete example, when a user opens a virtual store app on their smartphone and points their face at the camera, the app's emotion engine recognizes "joy." As a result, the next shopping screen prioritizes displaying new products and promotional information that are of interest to the user.

[1529] Examples of prompt statements are as follows:

[1530] "By analyzing the user's facial expression, if an emotion of 'joy' is detected, we will suggest products that are suitable for that user."

[1531] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1532] Step 1:

[1533] The server retrieves data including descriptions and image data. It uses an API to collect product data from virtual stores. The input is the API endpoint URL, and the output is a list of product descriptions and image URLs.

[1534] Step 2:

[1535] The server extracts text information from the acquired image data using optical character recognition (OCR) technology. Specifically, it analyzes the image data using the Google Cloud Vision API and extracts the text information. The input is image data, and the output is the extracted text data.

[1536] Step 3:

[1537] The server stores the analyzed text information and explanatory text in a database. The database stores data using unique identifiers. Input consists of text information, explanatory text, and a unique identifier, while output is the verification result of the stored data.

[1538] Step 4:

[1539] The user requests specific product information using a terminal. The user's request is sent to the server. The input is the user's request content (such as the product ID), and the output is the result of sending the request to the server.

[1540] Step 5:

[1541] The server searches the database based on the user's request and retrieves the relevant product data. The input is the user's request, and the output is the relevant product data.

[1542] Step 6:

[1543] The server sends the acquired product data to the user's terminal. The input is the acquired product data, and the output is the result of the transmission to the user's terminal.

[1544] Step 7:

[1545] The device acquires sensor data in real time from its built-in camera and microphone, and transmits this data to the emotion engine. The input is the sensor data from the camera and microphone, and the output is the result of the data transmission to the emotion engine.

[1546] Step 8:

[1547] The emotion engine uses the Google Cloud Vision API and the Hugging Face Transformers library to analyze the user's emotional state from their facial expressions and voice. The input is sensor data, and the output is the recognized emotional state.

[1548] Step 9:

[1549] Based on the recognized emotional state, the server dynamically optimizes the information it displays. Specifically, it adjusts product information and suggestions according to the user's emotions. The input is emotional state data, and the output is optimized product information.

[1550] Step 10:

[1551] The server sends optimized product information to the user's terminal, which then displays it visually. The input is the optimized product information, and the output is the visual display result provided to the user.

[1552] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1553] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1554] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1555] [Fourth Embodiment]

[1556] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1557] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1558] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1559] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1560] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1561] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1562] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1563] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1564] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1565] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1566] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1567] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1568] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1569] This invention relates to a system that collects data, including explanatory text, and image data from museums and exhibition facilities, analyzes them, and provides them in a user-accessible format. This system operates with the server, terminal, and user elements working in coordination with each other as follows.

[1570] 1. Data Collection

[1571] The server retrieves descriptive text and image data about the exhibits using the museum's Web API and other data provision methods.

[1572] For example, the server retrieves a list of exhibits and their details by calling the museum's API, GET / api / exhibits.

[1573] 2. Data Analysis and Text Conversion

[1574] The server analyzes the acquired image data of the exhibits and extracts text information from the images using optical character recognition (OCR) technology.

[1575] For example, extract the textual information "History of the Pharaohs" from images of exhibits at the "Ancient Egypt Exhibition."

[1576] 3. Saving to the database

[1577] The server stores the analyzed text information and explanatory data, along with a unique identifier, in a database. This allows for a quick response to user search requests.

[1578] As an example, data on "History of the Pharaohs" associated with ID "AE001" will be saved to the database.

[1579] 4. Processing user requests

[1580] Users use their devices to request information about specific exhibits. For example, a request is sent to the server by accessing a specific URL through the device's browser.

[1581] The server receives this request and searches for the relevant data in the database.

[1582] 5. Data transmission and display

[1583] The server sends the searched data to the user's device. The transmitted data includes descriptions, text information, and related images.

[1584] The terminal visually displays the received data to the user. The user can then review the displayed information and understand the details of the exhibits.

[1585] Specific example

[1586] 1. Examples of data collection

[1587] The server calls the API GET / api / exhibits to retrieve information about the exhibits. For example, information about the "Ancient Egypt Exhibition" is returned as a response. This response includes the exhibit ID, name, description, and image URL.

[1588] The server uses the image URL to download images of the exhibits.

[1589] 2. Examples of data analysis and text conversion

[1590] The server performs OCR processing on the downloaded image. For example, it converts the text information "History of the Pharaohs" within the image into text.

[1591] 3. Example of saving to a database

[1592] The textual information titled "History of the Pharaohs" will be stored in the database, associated with exhibit ID "AE001." This will allow for quick access during future searches.

[1593] 4. Example of processing user requests

[1594] The user sends a request to the server by accessing a URL containing the specific exhibit ID "AE001" from their device's browser.

[1595] 5. Examples of data transmission and display

[1596] The server searches the database for information related to "AE001" and sends it to the user's terminal. The terminal then displays the data, including descriptions and images, to the user.

[1597] Thus, the system of the present invention efficiently collects, analyzes, and stores detailed information about exhibits, making it readily accessible to users, thereby ensuring that information about exhibits is provided fairly to people who have difficulty visiting in person.

[1598] The following describes the processing flow.

[1599] Step 1:

[1600] Data collection (server)

[1601] The server sends a GET request to the museum's Web API, named / api / exhibits, to retrieve a list of exhibits and detailed information about them.

[1602] The received response includes the exhibit ID, name, description, and image URL.

[1603] For example, retrieve {"id": "123", "name": "Ancient Egypt Exhibition", "description": "History of the Pharaohs", "image_url": "https: / / example.com / image1.jpg"}.

[1604] Step 2:

[1605] Image data acquisition (server)

[1606] The server uses the image URL in the response to send a GET / image1.jpg request to download the image data.

[1607] The image data file image1.jpg is retrieved and temporarily saved for later analysis.

[1608] Step 3:

[1609] OCR processing (server)

[1610] The server begins analyzing the acquired image data using optical character recognition (OCR) technology.

[1611] Extract text information from an image and save it as text data. Example: Extract the text "History of the Pharaohs" from the image image1.jpg.

[1612] Step 4:

[1613] Saving to the database (server)

[1614] The server stores the extracted text information and explanatory data, along with a unique identifier, in a database.

[1615] Example: Save the data for "History of the Pharaohs" associated with ID "AE001" to the database in the following format:

[1616] SQL

[1617] INSERT INTO exhibits (id, name, description, ocr_text, image_url)

[1618] VALUES ('AE001', 'Ancient Egypt Exhibition', 'History of the Pharaohs', 'History of the Pharaohs', 'https: / / example.com / image1.jpg');

[1619] Step 5:

[1620] Processing user requests (user, device, server)

[1621] The user uses a terminal to request information about a specific exhibit. For example, they might send a request to GET / api / exhibits / AE001.

[1622] The terminal sends a request to the server, and the server receives this request.

[1623] Step 6:

[1624] Execution of database queries (server)

[1625] The server searches the database for information on the relevant exhibit based on the received request.

[1626] Example: Execute an SQL query to retrieve exhibit data for ID "AE001":

[1627] SQL

[1628] SELECT FROM exhibits WHERE id = 'AE001';

[1629] The acquired data is formatted and converted into a format that can be sent back to the user's terminal.

[1630] Step 7:

[1631] Data transmission (server, terminal)

[1632] The server sends the formatted data to the user's terminal.

[1633] example:

[1634] json

[1635] {

[1636] "id": "AE001",

[1637] "name": "Ancient Egypt Exhibition",

[1638] "description": "History of the Pharaohs",

[1639] "ocr_text": "History of the Pharaohs",

[1640] "image_url": "https: / / example.com / image1.jpg"

[1641] }

[1642] Step 8:

[1643] Data display (terminal, user)

[1644] The terminal processes the received data to display it visually.

[1645] Users can view the explanatory text, OCR-generated text information, and related images displayed on the device screen.

[1646] This process allows users to efficiently and quickly obtain detailed information about exhibits without having to physically visit the museum.

[1647] (Example 1)

[1648] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1649] To provide timely and accurate information about exhibits in museums and exhibition facilities, a system is needed to effectively collect, analyze, and store explanatory texts and image data. However, conventional methods have low accuracy in analyzing collected data, making it difficult to respond quickly to user requests. Furthermore, there is a lack of means to fairly provide information about exhibits to people who have difficulty visiting in person. Therefore, the development of a system that enables efficient and highly accurate data collection, analysis, storage, and provision is necessary.

[1650] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1651] In this invention, the server includes means for acquiring data and image data, including descriptive text related to museums and exhibition facilities; means for extracting character information from image data using optical character recognition technology; means for storing the extracted character information and descriptive text in a database along with a unique identifier; means for searching the database for stored data based on a user request and transmitting the acquired data to the user; and means for displaying the data transmitted to the user. This enables the efficient collection, analysis, and storage of information on exhibits in museums and exhibition facilities, allowing users to access it quickly and providing information fairly to people who have difficulty visiting.

[1652] A museum is a facility for collecting, preserving, and displaying historical, cultural, and artistic exhibits.

[1653] An "exhibition facility" is a place where people can view exhibits for educational or entertainment purposes.

[1654] An "explanatory text" is a piece of writing that provides information and explanations about an exhibit.

[1655] "Data" refers to a collection of information or signals, and in this invention, it includes explanatory text and image data.

[1656] "Image data" refers to digital data that includes visual information such as photographs and illustrations of exhibits.

[1657] "Means of acquisition" refers to the methods and techniques used to collect the necessary data.

[1658] "Means of analysis" refers to methods and techniques for processing acquired data and extracting necessary information.

[1659] "Optical character recognition technology" is a technology that converts characters in an image into digital text.

[1660] "Textual information" refers to characters and sentences that are represented as digital text.

[1661] A "unique identifier" is a unique number or code used to independently identify data.

[1662] A "database" is a system for efficiently storing and managing large amounts of data, and for searching and extracting it as needed.

[1663] "Means of storage" refers to methods and technologies for recording data in storage devices such as databases.

[1664] A "request" refers to the operation or content of a request that a user makes to a system to request information.

[1665] "Means of searching" refers to methods and techniques for finding data within a database.

[1666] "Means of transmission" refers to the methods and technologies used to send data to a specified destination.

[1667] "Means of display" refers to methods and technologies for visually showing data on a device.

[1668] A "user" refers to an individual or organization that uses this system to access information.

[1669] This invention is a system that collects and analyzes explanatory texts and image data of exhibits in museums and exhibition facilities, and provides them in a user-accessible format. This system operates with the server, terminal, and user elements working in coordination with each other. Specific embodiments are described below.

[1670] Data collection methods

[1671] Server operation

[1672] The server calls Web APIs provided by museums and exhibition facilities to retrieve descriptive text and image data about the exhibits. Specifically, the server sends HTTP requests and receives responses in JSON format. The hardware used for this is a cloud service (e.g., AWS EC2). The software used is the Python language and an HTTP request library (e.g., requests).

[1673] Data analysis and text format

[1674] Server operation

[1675] The server analyzes the acquired image data and extracts text information from the image using optical character recognition (OCR) technology. This process utilizes APIs such as Google Cloud Vision. The server sends the image data to this API and organizes the returned text data.

[1676] Method of saving to the database

[1677] Server operation

[1678] The server stores the parsed text information and descriptions along with unique identifiers in a database. MySQL or PostgreSQL are used for this database. The server inserts the text data and exhibit IDs into the database tables.

[1679] User request processing methods

[1680] User actions

[1681] The user requests information about a specific exhibit using their device's browser. Specifically, they access a URL containing the ID of the specific exhibit. For example, they might access "http: / / example.com / exhibit?ID=AE001".

[1682] Server operation

[1683] The server receives a user request and searches for relevant data in the database. It executes an SQL query (e.g., "SELECT FROM exhibits WHERE id='AE001'") to retrieve the search results.

[1684] Data transmission and display methods

[1685] Server operation

[1686] The server sends the search results to the user's device in JSON format. The transmitted data includes descriptions of the exhibits, extracted text information, and associated images.

[1687] Terminal operation

[1688] The terminal analyzes the received data and generates HTML for display. The terminal displays the generated HTML in a browser, allowing the user to view detailed information about the exhibits.

[1689] Specific example

[1690] 1. Examples of data collection

[1691] The server retrieves information about the exhibits by calling "GET / api / exhibits". For example, information about the "Ancient Egypt Exhibition" is returned as a response. This response includes the exhibit ID, name, description, and image URL.

[1692] 2. Examples of data analysis and text conversion

[1693] The server performs OCR processing on the downloaded image, converting the text information "History of the Pharaohs" into text.

[1694] 3. Example of saving to a database

[1695] The textual information titled "History of the Pharaohs" will be stored in the database, associated with exhibit ID "AE001." This will allow for quick access during future searches.

[1696] 4. Example of processing user requests

[1697] The user accesses a URL containing the specific exhibit ID "AE001" from their device's browser and sends a request to the server.

[1698] 5. Examples of data transmission and display

[1699] The server searches the database for information related to "AE001" and sends it to the user's terminal. The terminal then displays the data, including descriptions and images, to the user.

[1700] Example of a prompt

[1701] Prompt: "Explain the important exhibits in the Ancient Egypt section, especially those related to the history of the pharaohs."

[1702] AI model response: "Information related to the history of the pharaohs shows how the kings of ancient Egypt ruled. Exhibit ID 'AE001' contains this information in detail."

[1703] The above describes a specific embodiment for carrying out this invention. This enables efficient and accurate collection, analysis, and storage of exhibit information, and allows users to access it quickly.

[1704] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1705] Step 1: Data Collection

[1706] Server operation

[1707] The server calls the Web API of the museum or exhibition facility to retrieve descriptive text and image data about the exhibits. First, the server sends an HTTP request to the API endpoint (e.g., "GET / api / exhibits"). The response is returned in JSON format. This JSON data includes the exhibit's ID, name, description, and image URL.

[1708] Input: API endpoint URL (e.g., "GET / api / exhibits")

[1709] Output: Exhibit data in JSON format

[1710] Specific actions:

[1711] The server sends an HTTP request: "GET / api / exhibits"

[1712] The server receives a response: data in JSON format.

[1713] The server extracts the exhibit's ID, name, description, and image URL from the response.

[1714] Step 2: Data Analysis and Text Conversion

[1715] Server operation

[1716] The server analyzes the acquired image data and extracts text information from the image using optical character recognition (OCR) technology. Using an OCR service such as the Google Cloud Vision API, the server sends the image and receives the text data.

[1717] Input: Downloaded image data

[1718] Output: Extracted text data

[1719] Specific actions:

[1720] The server sends image data to the Google Cloud Vision API.

[1721] The API recognizes text within an image and returns the text data.

[1722] The server organizes the returned text data into something like "The History of the Pharaohs."

[1723] Step 3: Saving to the database

[1724] Server operation

[1725] The server stores the parsed text information and explanatory text along with a unique identifier (ID) in a database. MySQL or PostgreSQL are used as the database system. Data is inserted via SQL queries.

[1726] Input: Parsed text data, unique identifier, description

[1727] Output: Records stored in the database

[1728] Specific actions:

[1729] The server inserts text data and exhibit IDs into the database using an SQL query.

[1730] Example: "INSERT INTO exhibits (id, name, description, text_data) VALUES ('AE001', 'History of the Pharaohs', 'Details about the Pharaohs', 'Text data in the image')"

[1731] Step 4: Processing user requests

[1732] User actions

[1733] The user requests information about a specific exhibit using their device's browser. Specifically, they access a URL that includes the exhibit ID (e.g., "http: / / example.com / exhibit?ID=AE001").

[1734] Server operation

[1735] The server parses the received request and searches for relevant data in the database. It then executes an SQL query to retrieve information about the requested exhibit.

[1736] Input: User request (URL)

[1737] Output: Exhibit data retrieved from the database

[1738] Specific actions:

[1739] The user accesses "http: / / example.com / exhibit?ID=AE001" in their browser.

[1740] The server receives the request and executes the SQL query "SELECT FROM exhibits WHERE id='AE001'".

[1741] The server retrieves the query results.

[1742] Step 5: Send and display data

[1743] Server operation

[1744] The server sends the search results to the user's device in JSON format. The transmitted data includes descriptions of the exhibits, extracted text information, and related images.

[1745] Terminal operation

[1746] The terminal analyzes the received data and generates HTML for display. The terminal then displays the generated HTML in a browser, allowing the user to view detailed information about the exhibits.

[1747] Input: Exhibit data (in JSON format) retrieved from the database.

[1748] Output: Exhibit information displayed visually to the user

[1749] Specific actions:

[1750] The server sends the search results to the terminal in JSON format.

[1751] The terminal parses the received JSON data and generates HTML.

[1752] The device displays the generated HTML in a browser, allowing the user to view detailed information about the exhibits.

[1753] The above outlines the specific processing steps of the system. This process allows users to efficiently access information about the exhibits.

[1754] (Application Example 1)

[1755] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1756] Currently, physical stores face the challenge of users not being able to easily obtain detailed product information and review data. In particular, small print on product labels or a lack of detailed information can reduce user purchasing intent. Furthermore, providing product information visually often requires additional effort from the store, making efficient information provision difficult. An efficient system is needed to solve these problems.

[1757] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1758] In this invention, the server includes means for acquiring data including descriptive text and image data; means for using optical character recognition technology to extract character information from the image data; means for storing the extracted character information and descriptive text in a database; means for searching the database for stored data based on a user request and transmitting the acquired data to the user; means for displaying the data transmitted to the user; and means for using a smartphone to acquire product identification information and display product descriptions. This allows users to easily obtain detailed information about products in physical stores and increase their purchasing intent. Furthermore, stores can provide product information efficiently, improving overall operational efficiency.

[1759] "Data including descriptive text" refers to text-based data that contains detailed information about exhibits or products.

[1760] "Image data" refers to data that includes photographs, diagrams, and other visual information of exhibits and products.

[1761] "Optical character recognition technology" is a technology that analyzes character information in an image and extracts it as text data.

[1762] A "database" is an information processing system for centrally storing and managing collected data.

[1763] A "unique identifier" is a unique ID number or code used to distinguish each piece of data from other pieces of data.

[1764] A "user request" is an access request made by a user seeking specific information.

[1765] "Searching" is the process of finding specific data from stored information.

[1766] "Sending" refers to the act of transferring the retrieved data to the user's device.

[1767] "Display" refers to the act of providing data visually to a user's device.

[1768] "Product identification information" refers to information such as barcodes and QR codes used to identify specific products.

[1769] A "smartphone" is a mobile phone-type information terminal used by users to obtain and display information.

[1770] To implement this invention, a system is required in which the server, terminal, and user elements work together in coordination. Below, a specific embodiment of this system, a product information provision system in a physical store, will be described.

[1771] Data collection

[1772] The server uses the store's API to retrieve product descriptions and image data. For example, it can retrieve data including product ID, product name, description, price, and image URL from the store's API. Furthermore, it can also collect product images in real time from cameras installed in the store.

[1773] Data analysis and text conversion

[1774] The server uses optical character recognition (OCR) technology to extract text information from collected product images. For example, it can retrieve information such as "organically grown" or "expiration date" from product packaging labels. The pytesseract library is used for this OCR processing.

[1775] Save to database

[1776] The server stores the extracted text information and descriptions in a database along with a unique identifier (product ID). This allows for quick data retrieval later. A database management system such as SQLite is used for the database.

[1777] Processing user requests

[1778] The user uses a smartphone app to scan product identification information, such as a QR code. The scanned information is sent to the server as a request. Based on this request, the server searches its database for the relevant product data.

[1779] Data transmission and display

[1780] The server sends the searched data to the user's smartphone. The transmitted data includes product names, descriptions, prices, images, ingredient information, and reviews. The user can visually confirm this information on their smartphone screen.

[1781] Specific example

[1782] Specific examples of data collection

[1783] The data retrieved from the store API is in the following format:

[1784] Product ID: "PRD001"

[1785] Product name: "Organic Green Tea"

[1786] Description: "100% organically grown, pesticide-free tea"

[1787] Price: 500 yen

[1788] Image URL: "https: / / store.com / images / greentea.jpg"

[1789] Specific examples of analysis using OCR

[1790] The server extracts the following text information from the product image:

[1791] "Organic cultivation"

[1792] "Best before date: December 2025"

[1793] Example of a prompt

[1794] Data collection:

[1795] response = requests.get('https: / / store-api.com / products')

[1796] OCR analysis:

[1797] text = pytesseract.image_to_string(Image.open('path / to / image.jpg'), lang='jpn')

[1798] In this way, the system according to this invention provides an environment in which users can easily obtain detailed information about products in physical stores, and enables stores to provide information efficiently.

[1799] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1800] Step 1:

[1801] Data collection

[1802] The server sends a request to the store's API to retrieve product descriptions and image data. Specifically, the server sends an HTTP GET request to the API endpoint and receives data in the response, including product ID, product name, description, price, and image URL. Input: API request, Output: Product data JSON.

[1803] Step 2:

[1804] Download image data

[1805] The server downloads product images based on the image URLs in the retrieved product data. Specifically, it sends an HTTP GET request to the image URL for each product and retrieves the image file as a response. Input: Image URL, Output: Image file.

[1806] Step 3:

[1807] Extraction of text information using OCR

[1808] The server extracts text information from downloaded image files using Optical Character Recognition (OCR) technology. Specifically, it uses the pytesseract library to extract text from image files. This text serves as supplementary information about the product. Input: Image file, Output: Extracted text.

[1809] Step 4:

[1810] Save to database

[1811] The server stores the extracted text information and retrieved product data in a database. Each data entry is assigned a unique identifier (product ID) for efficient management. Specifically, data is registered in an SQLite database using INSERT statements. Input: product data and extracted text; Output: database records.

[1812] Step 5:

[1813] User scan of product identification information

[1814] Users use a smartphone app to scan product identification information, such as QR codes, within the store. Specifically, they use their smartphone's camera to read QR codes, and the scanned data is displayed within the app. Input: QR code, Output: Product identification information.

[1815] Step 6:

[1816] Sending a user request

[1817] The user sends product identification information to the server as a request through the app. Specifically, the smartphone app sends an HTTP GET request containing the product ID to the server. Input: Product identification information, Output: Request sent.

[1818] Step 7:

[1819] Search from database

[1820] The server searches the database based on the product identification information received from the user and retrieves relevant product information. Specifically, it uses a SELECT statement to search the database using the product ID. Input: Product identification information, Output: Product data.

[1821] Step 8:

[1822] Sending product data to user terminals

[1823] The server sends the retrieved product data to the user's smartphone. Specifically, it sends the retrieved data as an HTTP response in JSON format. Input: Product data, Output: JSON response.

[1824] Step 9:

[1825] Displaying product information

[1826] The user visually displays product information received via a smartphone app. Specifically, the app displays product name, description, price, ingredient information, reviews, etc., on the screen. Input: JSON response, Output: Display of product information.

[1827] In this way, the server, terminal, and user work together at each step, making it possible for users to easily obtain product information in physical stores. This allows users to make purchasing decisions based on more detailed information, and stores can provide information more efficiently.

[1828] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1829] This invention combines a system that collects data including explanatory text and image data from museums and exhibition facilities, analyzes them, and provides them in a user-accessible format with an emotion engine that recognizes user emotions. This emotion engine makes it possible to dynamically optimize the information displayed according to the user's emotional state. Furthermore, it can learn from emotional data and provide information tailored to the user's preferences.

[1830] 1. Data Collection

[1831] The server retrieves descriptive text and image data about the exhibits using the museum's Web API and other data provision methods.

[1832] For example, the server retrieves a list of exhibits and their details by calling the museum's API, GET / api / exhibits.

[1833] 2. Acquisition of image data

[1834] The server uses the image URL in the response to send a GET / image1.jpg request and download the image data.

[1835] The image data file image1.jpg is retrieved and temporarily saved for later analysis.

[1836] 3. OCR processing

[1837] The server begins analyzing the acquired image data using optical character recognition (OCR) technology.

[1838] Extract text information from an image and save it as text data. Example: Extract the text "History of the Pharaohs" from the image image1.jpg.

[1839] 4. Saving to the database

[1840] The server stores the analyzed text information and explanatory data, along with a unique identifier, in a database.

[1841] Example: Save the data for "History of the Pharaohs" associated with ID "AE001" to the database in the following format:

[1842] SQL

[1843] INSERT INTO exhibits (id, name, description, ocr_text, image_url)

[1844] VALUES ('AE001', 'Ancient Egypt Exhibition', 'History of the Pharaohs', 'History of the Pharaohs', 'https: / / example.com / image1.jpg');

[1845] 5. Processing user requests

[1846] Users use their devices to request information about specific exhibits. For example, a request is sent to the server by accessing a specific URL through the device's browser.

[1847] The terminal sends a request to the server, and the server receives this request.

[1848] 6. Executing database queries

[1849] The server searches the database for information on the relevant exhibit based on the received request.

[1850] Example: Execute an SQL query to retrieve exhibit data for ID "AE001":

[1851] SQL

[1852] SELECT FROM exhibits WHERE id = 'AE001';

[1853] The acquired data is formatted and converted into a format that can be sent back to the user's terminal.

[1854] 7. Emotion recognition by an emotion engine

[1855] The server uses cameras and sensors installed on the user's device to recognize the user's emotional state in real time.

[1856] The emotion engine uses facial recognition and voice analysis technologies to extract emotions from the user's facial expressions and speech.

[1857] The recognized emotional state (e.g., joy, interest, surprise, confusion, etc.) is stored as data.

[1858] 8. Data transmission and display

[1859] The server dynamically optimizes the data it displays based on the emotional state recognized by the emotion engine.

[1860] The optimized data, including the exhibit descriptions, OCR text information, and related images, is sent to the user's device.

[1861] The terminal processes the received data to display it visually and then provides it to the user.

[1862] 9. Learning from user sentiment data

[1863] The server stores user emotion data recognized by the emotion engine and learns the user's preferences and interests.

[1864] The learned data will be used to provide information in the future. For example, if there is new information related to exhibits that the user has shown interest in in the past, the system will be adjusted to prioritize displaying that information.

[1865] Specific example

[1866] 1. Examples of data collection

[1867] The server calls the API GET / api / exhibits to retrieve information about the exhibits. For example, information about the "Ancient Egypt Exhibition" is returned as a response. This response includes the exhibit ID, name, description, and image URL.

[1868] The server uses the image URL to download images of the exhibits.

[1869] 2. Examples of data analysis and text conversion

[1870] The server performs OCR processing on the downloaded image. For example, it converts the text information "History of the Pharaohs" within the image into text.

[1871] 3. Examples of optimization using an emotion engine

[1872] When a user requests information about "AE001" and the device receives data from the server, the server analyzes the user's camera footage and recognizes that the user is in an "excited" state.

[1873] The server sends optimized data, including more detailed information and additional images, to users in an "excited" state.

[1874] 4. Example of data display

[1875] The device visually displays and provides optimized data to the user. For example, it might show a detailed description of "The History of the Pharaohs" along with several related images.

[1876] In this way, the system of the present invention can efficiently collect, analyze, and store detailed information about exhibits, making it readily accessible to users, and can also optimize the information according to the user's emotional state, providing a personalized experience.

[1877] The following describes the processing flow.

[1878] Step 1:

[1879] Data collection (server)

[1880] The server sends a GET request to the museum's Web API, named / api / exhibits, to retrieve a list of exhibits and detailed information about them.

[1881] The received response includes the exhibit ID, name, description, and image URL.

[1882] For example, retrieve {"id": "123", "name": "Ancient Egypt Exhibition", "description": "History of the Pharaohs", "image_url": "https: / / example.com / image1.jpg"}.

[1883] Step 2:

[1884] Image data acquisition (server)

[1885] The server uses the image URL in the response to send a GET / image1.jpg request to download the image data.

[1886] The image data file image1.jpg is retrieved and temporarily saved for later analysis.

[1887] Step 3:

[1888] OCR processing (server)

[1889] The server begins analyzing the acquired image data using optical character recognition (OCR) technology.

[1890] This process extracts text information from an image and saves it as text data. For example, it extracts the text "History of the Pharaohs" from the image image1.jpg.

[1891] Step 4:

[1892] Saving to the database (server)

[1893] The server stores the analyzed text information and explanatory data, along with a unique identifier, in a database.

[1894] As an example, the data for "History of the Pharaohs" associated with ID "AE001" will be saved to the database in the following format:

[1895] SQL

[1896] INSERT INTO exhibits (id, name, description, ocr_text, image_url)

[1897] VALUES ('AE001', 'Ancient Egypt Exhibition', 'History of the Pharaohs', 'History of the Pharaohs', 'https: / / example.com / image1.jpg');

[1898] Step 5:

[1899] Processing user requests (user, device, server)

[1900] The user uses their device to request information about a specific exhibit. For example, they might send a request like GET / api / exhibits / AE001 from their device's browser.

[1901] The server receives this request and prepares the corresponding data.

[1902] Step 6:

[1903] Execution of database queries (server)

[1904] Based on the request, the server searches the database for information about the relevant exhibit.

[1905] For example, execute an SQL query to retrieve exhibit data for ID "AE001":

[1906] SQL

[1907] SELECT FROM exhibits WHERE id = 'AE001';

[1908] The acquired data is formatted and converted into a format that can be sent back to the user's terminal.

[1909] Step 7:

[1910] Emotion recognition using an emotion engine (server, terminal)

[1911] The device uses cameras and sensors to capture the user's facial expressions and voice, and sends this information to the emotion engine.

[1912] The server uses an emotion engine to analyze and recognize the user's emotional state in real time.

[1913] For example, the emotion engine recognizes that a user is in a state of "excitement" after viewing the exhibit's description.

[1914] Step 8:

[1915] Data transmission and display (server, terminal, user)

[1916] The server dynamically optimizes the data it displays based on the recognized emotional state.

[1917] The optimized data, including the exhibit descriptions, OCR text information, and related images, is sent to the user's device.

[1918] The terminal visually displays the received data, allowing users to view detailed information about the exhibits.

[1919] Step 9:

[1920] Learning user sentiment data (server)

[1921] The server stores user emotion data recognized by the emotion engine and learns the user's preferences and interests.

[1922] For example, the system could be adjusted to prioritize displaying information related to exhibits the user has previously shown interest in during subsequent requests.

[1923] This process allows the system to provide information that responds to the user's real-time emotions, delivering a personalized experience.

[1924] (Example 2)

[1925] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1926] Traditional museum and exhibition information systems have faced challenges in maintaining user interest because the information provided is not dynamically optimized according to the user's emotional state or individual interests. Furthermore, the lack of consideration for user emotional states hinders the quality of understanding and experience of exhibits. Additionally, inefficient management and retrieval of acquired data, resulting in a mix of non-uniquely identifiable data, makes it difficult to quickly provide necessary information.

[1927] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1928] In this invention, the server includes means for acquiring data including descriptive text and image data; means for extracting character information from the acquired image data using optical character recognition technology; means for storing the extracted character information and descriptive text in a database; means for searching the database for stored data based on a user request and transmitting the acquired data to the user; means for recognizing the user's emotional state; means for dynamically optimizing the data to be displayed based on the recognized emotional state; and means for visually displaying the data on the user terminal. This enables personalized information provision according to the user's emotional state and interests, thereby improving the quality of the user experience. Furthermore, it enables efficient data management and retrieval, and allows for the rapid provision of uniquely identified data.

[1929] "Data including descriptive text" refers to data that contains descriptive information in text format about exhibits and related information.

[1930] "Image data" refers to data in image format that visually represents exhibits and related information.

[1931] "Optical character recognition technology" is a technology for extracting character information from image data, and is also known as OCR (Optical Character Recognition).

[1932] "Character information" refers to information in text format extracted using optical character recognition technology.

[1933] A "database" is a digital system for efficiently storing, managing, and retrieving acquired and analyzed data.

[1934] A "user request" is a request that a user sends to the system seeking specific information.

[1935] "User emotional state" refers to information about the user's current mental and emotional condition, and is recognized from facial expressions, voice analysis, and other sources.

[1936] "Dynamic optimization" means adjusting the information displayed in real time according to the user's current state and situation, and providing it in the most optimal form.

[1937] "Visually displaying" means displaying information graphically on the user's device, and it means providing images and text in a format that can be visually confirmed.

[1938] A "unique identifier" is a unique identifier assigned to each piece of data to distinguish it from other pieces of data.

[1939] This invention relates to a system for collecting, analyzing, and storing detailed information about exhibits in museums and exhibition facilities, and providing personalized information to users. This system acquires data including descriptive text and image data, analyzes them, and provides them in a user-accessible format. Furthermore, it can recognize user emotions and dynamically optimize the information. Specific embodiments are described below.

[1940] Hardware and software to be used

[1941] This system consists of three main elements: a server, terminals, and users. The server is a computer system with a high-performance processor and sufficient memory, which collects and analyzes data. It also uses an SQL-based database (e.g., PostgreSQL) as its database system. For image data analysis, it uses Tesseract, a widely used optical character recognition (OCR) technology. For emotion recognition, it uses an emotion analysis module called EmotionEngine. User terminals include smartphones, tablets, and personal computers equipped with cameras and sensors.

[1942] Data collection and analysis

[1943] The server calls the museum's API to retrieve descriptive text and image data about the exhibits. The retrieved image data is converted into text information using optical character recognition (OCR) technology. For example, it calls the API GET / api / exhibits, obtains the image URL from the response, accesses the corresponding URL to download the image, and extracts text information from that image. Through this process, the text data "History of the Pharaohs" is extracted from the image image1.jpg.

[1944] Data storage

[1945] The server stores the extracted text information and descriptions in the database, associating them with a unique identifier (e.g., ID "AE001"). For example, data related to the "Ancient Egypt Exhibition" is stored in the database in the following format:

[1946] Exhibit ID: AE001

[1947] Exhibition Title: Ancient Egypt Exhibition

[1948] Description: History of the Pharaohs

[1949] OCR text: History of the Pharaohs

[1950] Image URL: https: / / example.com / image1.jpg

[1951] Processing and optimizing user requests

[1952] Users request information about specific exhibits using their devices. For example, accessing a specific URL from a browser sends a request to the server. The server receives this request, searches its database for information about the relevant exhibit, and sends it back to the user. Furthermore, the system recognizes the user's emotional state in real time through the device's camera and sensors, and extracts it using an emotion engine. Based on the recognized emotional state (e.g., "excited"), the displayed data is dynamically optimized. For example, if the user is in an "excited" state, data containing more detailed information and additional images is sent to the user's device. The device then visually displays the optimized information based on this.

[1953] Learning user sentiment data

[1954] The server stores user emotion data recognized by the emotion engine and learns the user's preferences and interests. This learned data is used to provide information in the future. For example, if there is new information related to an exhibit that the user has shown interest in in the past, that information will be displayed preferentially.

[1955] In this way, the system of the present invention can collect, analyze, and store detailed information about exhibits, making it readily accessible to users, and can also optimize the information according to the user's emotional state to provide a personalized experience.

[1956] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1957] Step 1:

[1958] Data collection

[1959] The server calls the museum's Web API to retrieve descriptive text and image data about the exhibits.

[1960] Input: API call endpoint (e.g., GET / api / exhibits)

[1961] Data processing: Analyze the JSON response data obtained from the API and extract a list of exhibits.

[1962] Output: Exhibit list (e.g., exhibit ID, name, description, image URL, etc.)

[1963] Step 2:

[1964] Acquisition of image data

[1965] The server extracts image URLs from the list of exhibits and uses those URLs to download the image data.

[1966] Input: Image URL from the exhibit list (e.g., https: / / example.com / image1.jpg)

[1967] Data processing: Send an HTTP request and receive the corresponding image data.

[1968] Output: Image data file (e.g., image1.jpg)

[1969] Step 3:

[1970] OCR processing

[1971] The server applies optical character recognition (OCR) technology to the downloaded image data to extract text information.

[1972] Input: Image data file (e.g., image1.jpg)

[1973] Data processing: Analyze the text in the image using OCR software (e.g., Tesseract).

[1974] Output: Extracted text data (e.g., "History of the Pharaohs")

[1975] Step 4:

[1976] Save to database

[1977] The server stores the extracted text information and descriptions in a database along with a unique identifier.

[1978] Input: Exhibit ID, Name, Description, OCR text, Image URL

[1979] Data processing: Execute SQL queries to insert information into the database.

[1980] Output: Database entry where the data is stored (e.g., information associated with ID "AE001")

[1981] Step 5:

[1982] Processing user requests

[1983] Users use their devices to request information about specific exhibits.

[1984] Input: Request URL sent by the user from their browser (e.g., https: / / museum-website.com / exhibit?id=AE001)

[1985] Data processing: The terminal sends a request to the server, and the server receives the request.

[1986] Output: Request information forwarded to the server

[1987] Step 6:

[1988] Execution of database queries

[1989] Based on the received request, the server searches the database for information about the relevant exhibit.

[1990] Input: Exhibit ID in the request (e.g., AE001)

[1991] Data processing: Execute SQL queries to search for the relevant data.

[1992] Output: Searched exhibit information (e.g., exhibit name, description, OCR text, image URL)

[1993] Step 7:

[1994] Emotion recognition by an emotion engine

[1995] The server uses cameras and sensors installed on the user's device to recognize the user's emotional state in real time.

[1996] Input: User's camera video and audio data

[1997] Data processing: Analyze emotional information using an emotion engine (e.g., EmotionEngine).

[1998] Output: Recognized emotional state (e.g., "excited")

[1999] Step 8:

[2000] Data transmission and display

[2001] The server dynamically optimizes the data displayed based on the recognized emotional state.

[2002] Input: Exhibit information, perceived emotional state

[2003] Data processing: Generate optimized information and send it to the user's terminal.

[2004] Output: Dynamically optimized exhibit information

[2005] Step 9:

[2006] Displaying data

[2007] The terminal visually displays the received data and provides it to the user.

[2008] Input: Optimization information sent from the server

[2009] Data processing: Converting data into a layout and format for display.

[2010] Output: Detailed exhibit information provided to the user visually (e.g., detailed description, images)

[2011] Step 10:

[2012] Learning user sentiment data

[2013] The server stores user emotion data recognized by the emotion engine and learns the user's preferences and interests.

[2014] Input: Recognized emotional state, corresponding exhibit information

[2015] Data processing: Emotional data and exhibit information are stored in a database and analyzed.

[2016] Output: Recommendations based on learned user preferences

[2017] (Application Example 2)

[2018] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2019] Conventional exhibit information systems lacked dynamic data optimization based on user emotional states, resulting in insufficient improvement in the quality of the user experience. Furthermore, they lacked personalized information tailored to user preferences, making it difficult to quickly and accurately provide information of interest to users. Additionally, conventional systems lacked the functionality to understand and adjust displayed content based on real-time user emotional states, thus failing to provide information that met user interests and concerns.

[2020] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[2021] In this invention, the server includes means for acquiring data including descriptive text and image data; means for analyzing the aforementioned data including descriptive text and image data and extracting text information from the image data; means for storing the extracted text information and descriptive text in a database; means for searching the database for stored data based on a user request and sending the acquired data to the user; means for displaying the data sent to the user; means for incorporating an emotion engine that analyzes sensor data acquired from a terminal to recognize the user's emotional state; and means for dynamically optimizing the information to be displayed according to the user's emotional state. This makes it possible to read the user's emotional state and provide information that corresponds to it in real time with dynamic optimization. Personalized information provision tailored to the user's interests and concerns can be realized, improving the quality of the user experience.

[2022] A "description" is text data that provides detailed information about an exhibit.

[2023] "Image data" refers to digital image files that include visual representations of exhibits and related items.

[2024] A "database" is an information system for systematically storing and managing descriptive text and image data.

[2025] A "user" or "device" refers to an individual requesting information about an exhibit and the device they use (e.g., a smartphone or tablet).

[2026] A "request" is a request from a user to retrieve information about a specific exhibit from a database.

[2027] An "emotion engine" is a system that recognizes the user's emotional state and dynamically optimizes the information displayed based on that data.

[2028] "Sensor data" refers to data such as the user's facial expressions and voice, acquired from the device's camera, microphone, etc.

[2029] "Dynamic optimization" is a process that adjusts the displayed information according to the user's real-time emotional state.

[2030] "Extraction" is the process of extracting text information from image data.

[2031] "Storage" refers to the process of recording and storing the analyzed data in a database.

[2032] This invention aims to improve the user experience in virtual stores. This system uses an emotion engine to recognize the user's emotional state and provides optimized product information tailored to that emotion. Specifically, the server implements this invention through the following main means.

[2033] The server first retrieves data including descriptions and image data. This data is collected from virtual stores via an API.

[2034] The server uses optical character recognition (OCR) technology to extract text information from the acquired image data. This allows for obtaining detailed text information about the product from the image data.

[2035] The parsed text information and explanatory text are stored in a database. The stored data is associated with a unique identifier, facilitating subsequent searching and retrieval.

[2036] The user requests specific product information using a device (e.g., a smartphone). In response to this request, the server searches the database, retrieves the relevant product data, and sends it to the user's device.

[2037] During this process, sensor data is acquired in real time from the camera and microphone installed in the device, and the emotion engine analyzes the user's emotional state. The emotion engine uses the Google Cloud Vision API and the Hugging Face Transformers library, among others, to recognize emotions from the user's facial expressions and voice.

[2038] The information displayed is dynamically optimized according to the recognized emotional state (e.g., "joy" or "surprise"). For example, if the emotion of "joy" is detected, the system will prioritize displaying new product and promotional information that the user is likely to be interested in.

[2039] To give a concrete example, when a user opens a virtual store app on their smartphone and points their face at the camera, the app's emotion engine recognizes "joy." As a result, the next shopping screen prioritizes displaying new products and promotional information that are of interest to the user.

[2040] Examples of prompt statements are as follows:

[2041] "By analyzing the user's facial expression, if an emotion of 'joy' is detected, we will suggest products that are suitable for that user."

[2042] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[2043] Step 1:

[2044] The server retrieves data including descriptions and image data. It uses an API to collect product data from virtual stores. The input is the API endpoint URL, and the output is a list of product descriptions and image URLs.

[2045] Step 2:

[2046] The server extracts text information from the acquired image data using optical character recognition (OCR) technology. Specifically, it analyzes the image data using the Google Cloud Vision API and extracts the text information. The input is image data, and the output is the extracted text data.

[2047] Step 3:

[2048] The server stores the analyzed text information and explanatory text in a database. The database stores data using unique identifiers. Input consists of text information, explanatory text, and a unique identifier, while output is the verification result of the stored data.

[2049] Step 4:

[2050] The user requests specific product information using a terminal. The user's request is sent to the server. The input is the user's request content (such as the product ID), and the output is the result of sending the request to the server.

[2051] Step 5:

[2052] The server searches the database based on the user's request and retrieves the relevant product data. The input is the user's request, and the output is the relevant product data.

[2053] Step 6:

[2054] The server sends the acquired product data to the user's terminal. The input is the acquired product data, and the output is the result of the transmission to the user's terminal.

[2055] Step 7:

[2056] The device acquires sensor data in real time from its built-in camera and microphone, and transmits this data to the emotion engine. The input is the sensor data from the camera and microphone, and the output is the result of the data transmission to the emotion engine.

[2057] Step 8:

[2058] The emotion engine uses the Google Cloud Vision API and the Hugging Face Transformers library to analyze the user's emotional state from their facial expressions and voice. The input is sensor data, and the output is the recognized emotional state.

[2059] Step 9:

[2060] Based on the recognized emotional state, the server dynamically optimizes the information it displays. Specifically, it adjusts product information and suggestions according to the user's emotions. The input is emotional state data, and the output is optimized product information.

[2061] Step 10:

[2062] The server sends optimized product information to the user's terminal, which then displays it visually. The input is the optimized product information, and the output is the visual display result provided to the user.

[2063] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[2064] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2065] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[2066] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2067] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. In the upper and lower directions of the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. Also, the upper side of the concentric circles is where "pleasant" emotions are located, and the lower side is where "unpleasant" emotions are located. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[2068] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[2069] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[2070] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[2071] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[2072] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[2073] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[2074] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[2075] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[2076] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2077] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[2078] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[2079] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[2080] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[2081] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[2082] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[2083] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[2084] The following is further disclosed regarding the embodiments described above.

[2085] (Claim 1)

[2086] A means of acquiring data and image data, including descriptive text.

[2087] A means for analyzing data and image data containing the above-mentioned explanatory text, and for extracting textual information from the image data,

[2088] A means for storing extracted text information and explanatory text in a database,

[2089] A means of searching for data stored in a database based on a user request and sending the retrieved data to the user,

[2090] A means of displaying data sent to the user,

[2091] A system that includes this.

[2092] (Claim 2)

[2093] The system according to claim 1, comprising means of using optical character recognition technology to extract character information from image data.

[2094] (Claim 3)

[2095] The system according to claim 1, comprising means for assigning a unique identifier to each data item in order to accurately identify and retrieve the acquired data.

[2096] "Example 1"

[2097] (Claim 1)

[2098] A means of acquiring data and image data, including descriptive text related to museums and exhibition facilities,

[2099] A means for analyzing data and image data containing the above-mentioned explanatory text, and for extracting character information from image data using optical character recognition technology,

[2100] A means for storing the extracted text information and explanatory text in a database along with a unique identifier,

[2101] A means of searching for data stored in a database based on a user request and sending the retrieved data to the user,

[2102] A means of displaying data sent to the user,

[2103] A system that includes this.

[2104] (Claim 2)

[2105] The system according to claim 1, which uses a web API to obtain data including descriptive text and image data.

[2106] (Claim 3)

[2107] The system according to claim 1, comprising means for assigning a unique identifier to each data item in order to accurately identify and retrieve the acquired data.

[2108] "Application Example 1"

[2109] (Claim 1)

[2110] A means of acquiring data and image data, including descriptive text.

[2111] A means for analyzing data and image data containing the above-mentioned explanatory text, and for extracting textual information from the image data,

[2112] A means for storing extracted text information and explanatory text in a database,

[2113] A means of searching for data stored in a database based on a user request and sending the retrieved data to the user,

[2114] A means of displaying data sent to the user,

[2115] A means of using a smartphone to obtain product identification information and display product descriptions,

[2116] A system that includes this.

[2117] (Claim 2)

[2118] The system according to claim 1, comprising means of using optical character recognition technology to extract character information from image data.

[2119] (Claim 3)

[2120] The system according to claim 1, comprising means for assigning a unique identifier to each data item in order to accurately identify and retrieve the acquired data.

[2121] "Example 2 of combining an emotion engine"

[2122] (Claim 1)

[2123] A means of acquiring data and image data, including descriptive text.

[2124] A means for extracting character information from acquired image data using optical character recognition technology,

[2125] A means for storing extracted text information and explanatory text in a database,

[2126] A means of searching for data stored in a database based on a user request and sending the retrieved data to the user,

[2127] A means of recognizing the user's emotional state,

[2128] A means for dynamically optimizing the data displayed based on the recognized emotional state,

[2129] A means of visually displaying data on the user terminal,

[2130] A system that includes this.

[2131] (Claim 2)

[2132] The system according to claim 1, comprising means of using optical character recognition technology to extract character information from image data.

[2133] (Claim 3)

[2134] The system according to claim 1, comprising means for assigning a unique identifier to each data item in order to accurately identify and retrieve the acquired data.

[2135] "Application example 2 when combining with an emotional engine"

[2136] (Claim 1)

[2137] A means of acquiring data and image data, including descriptive text.

[2138] A means for analyzing data and image data containing the above-mentioned explanatory text, and for extracting textual information from the image data,

[2139] A means for storing extracted text information and explanatory text in a database,

[2140] A means of searching for data stored in a database based on a user request and sending the retrieved data to the user,

[2141] A means of displaying data sent to the user,

[2142] A means of incorporating an emotion engine that analyzes sensor data acquired from the device in order to recognize the user's emotional state,

[2143] A means for dynamically optimizing the information displayed according to the user's emotional state,

[2144] A system that includes this.

[2145] (Claim 2)

[2146] The system according to claim 1, comprising means of using optical character recognition technology to extract character information from image data.

[2147] (Claim 3)

[2148] The system according to claim 1, comprising means for assigning a unique identifier to each data item in order to accurately identify and retrieve the acquired data. [Explanation of symbols]

[2149] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of acquiring data and image data, including descriptive text. A means for analyzing data and image data containing the above-mentioned explanatory text, and for extracting textual information from the image data, A means for storing extracted text information and explanatory text in a database, A means of searching for data stored in a database based on a user request and sending the retrieved data to the user, A means of displaying data sent to the user, A system that includes this.

2. The system according to claim 1, comprising means of using optical character recognition technology to extract character information from image data.

3. The system according to claim 1, comprising means for assigning a unique identifier to each data item in order to accurately identify and retrieve the acquired data.

Images