System

Abstract

A system is provided.SOLUTION: A system comprising: means for digitally archiving museum exhibits; means for displaying the digitized exhibits on a user terminal; means for recommending an optimal tour guide based on the user's interests and concerns; means for conducting an online tour in which the guide describes the exhibits; and means for supporting real-time interactive communication between the guide and the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Geographical and time constraints are major problems in modern art museum operations. Visiting a museum is difficult, especially for people who live far away or are busy, limiting opportunities to appreciate art. Museums are also seeking new revenue sources to improve revenue and ensure sustainable operations. However, traditional methods make it difficult to promote digitalization and improve the quality of art appreciation from remote locations. Furthermore, there is a need to provide effective explanations of the exhibits and offer personalized experiences tailored to the interests of visitors. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by developing a system that digitally archives museum exhibits and provides a virtual museum that can be accessed online. Specifically, the system provides a means for digitizing museum exhibits, a means for displaying the digitized exhibits on a user's device, a means for recommending the most suitable tour guide based on the user's interests, a means for conducting online tours, and a means for supporting real-time two-way communication between guides and users. The system also includes a means for converting and translating the guide's commentary into text and providing it to the user, and a means for collecting user feedback after the tour and providing it to the museum, thereby improving the user experience and streamlining museum operations.

[0006] A "museum" is a facility that houses artworks and exhibits and is open to the public.

[0007] "Exhibits" are works of art, documents, etc. that are stored in a museum and displayed for the purpose of being made public.

[0008] A "digital archive" is a collection of paper media or physical exhibits that have been digitized and stored as electronic data.

[0009] "Users" are people who use the system to view museum exhibits and take online tours.

[0010] A "user terminal" is a device such as a computer, smartphone, or tablet that a user uses to access the system.

[0011] A "tour guide" is an individual with professional or specialized knowledge whose role is to provide an interpretation of exhibits.

[0012] An "online tour" is a tour of a museum exhibit that takes place in real time over the Internet.

[0013] "Two-way communication" is a form of communication that allows for the mutual exchange of information and opinions between a speaker and a listener.

[0014] "Text conversion" is the process of converting audio or video content into text data.

[0015] "Translation" is the process of converting content written in one language into another language.

[0016] "Feedback" refers to the impressions and opinions received from users after the tour has ended, and analyzing these will help improve the service. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention provides a system that digitally archives art museum exhibits and provides a virtual museum that users can access online. Specific embodiments of each function are described below.

[0039] 1. Digital archiving of museums

[0040] server

[0041] The server receives digital data (images, videos, text, etc.) of exhibits provided by the museum and temporarily stores it. For example, when the museum uploads a high-resolution image of "Famous Painting A," the server receives this data and stores it in a temporary storage directory. The server then analyzes the digital data and extracts metadata (e.g., artist name, year of production, technique used). The server then converts the data into an appropriate format and stores it in a dedicated database. The server also indexes the data to enable quick searches and filtering later.

[0042] 2. Online tour guide matching

[0043] user

[0044] A user accesses the virtual museum from their device and requests a tour guide. For example, if they input their desire to "learn about Impressionism," this information is sent to the server. The server analyzes this request and searches for guide information in its database. For example, it creates a list of guides who are knowledgeable about "Impressionism" and sends this list to the user. The user selects the desired guide from the displayed guides and sends that information to the server. The server then requests confirmation and approval of the tour date and time from the selected guide, and if approval is granted, sends the confirmed information to the user.

[0045] 3. Conducting Virtual Tours

[0046] Server, user terminal, guide terminal

[0047] When the tour begins, the server sends a notification to the user and guide's devices that the tour is about to begin, and pre-loads the digital data necessary for the tour. The guide uses a dedicated application to operate the exhibits while giving their explanation. At this time, the guide's device sends the explanation and screen operation information to the user in real time via the server. Users can operate the virtual space on their own devices and view the exhibits in detail. Users can also send questions using the chat function. The server receives questions from users, converts them to text, and sends them to the guide. These questions are translated as needed, making multilingual support possible. The guide answers the users' questions in real time.

[0048] 4. Secondary Features of the Service

[0049] Server, user terminal

[0050] After the tour ends, the server displays a survey on the user's device and collects responses. The collected survey results and comments made during the tour are analyzed, and a feedback report is generated and provided to the museum. This allows the museum to improve its services. Furthermore, based on the exhibits and artists that the user showed interest in during the tour, the server recommends products from the museum store and provides a purchasing link to the user's device. For example, if a user is interested in a replica of "Masterpiece A," related products will be displayed and a link to purchase them will be provided.

[0051] This will provide museums with new revenue streams and allow visitors to have a high-quality guided museum experience remotely.

[0052] The processing flow will be explained below.

[0053] 1. Digital archiving of museums

[0054] Step 1:

[0055] Server: Provides a dedicated upload form to receive digital data (images, videos, text, etc.) of exhibits provided by the museum. When a museum staff member uploads a high-resolution image of "Masterpiece A," the server receives this data and moves it to a directory for temporary storage.

[0056] Step 2:

[0057] Server: Analyzes the received digital data and runs analysis scripts to automatically extract metadata (artist name, year of production, technique, etc.) Once the metadata is extracted, the server stores this data in a digital archive database.

[0058] Step 3:

[0059] Server: Indexes the data to enable quick searches and filtering later. The indexed data is efficiently managed in a database.

[0060] 2. Online tour guide matching

[0061] Step 1:

[0062] Device: The user accesses the virtual museum website from their device, enters the tour guide request screen, and enters "I would like a tour of the Impressionists." This information is sent to the server.

[0063] Step 2:

[0064] Server: Analyzes the user's request, searches for guide information in the database, generates a list of target guides, and sends it to the user's device.

[0065] Step 3:

[0066] Terminal: The user selects the desired guide from the displayed guides and sends the selected information to the server. For example, if "Guide B" is selected, the information is sent to the server.

[0067] Step 4:

[0068] Server: Requests confirmation and approval of the tour date and time from the selected guide, and sends the confirmation information to the user once approval is received.

[0069] 3. Conducting Virtual Tours

[0070] Step 1:

[0071] Server: When the tour start time arrives, the server sends a tour start notification to the user and guide's devices and preloads the digital data required for the tour.

[0072] Step 2:

[0073] Guide: The guide uses a dedicated application to operate the exhibits and provide commentary. Screen operations and commentary are distributed to users in real time via a server.

[0074] Step 3:

[0075] Users: Users can operate the virtual space on their own devices, view the exhibits in detail, and send questions to the guide using the chat function.

[0076] Step 4:

[0077] Server: Converts user questions into text and sends it to the guide. If necessary, translates the questions to support multiple languages.

[0078] Step 5:

[0079] Guide: The guide answers users' questions in real time.

[0080] 4. Secondary Features of the Service

[0081] Step 1:

[0082] Terminal: After the tour ends, the server displays a questionnaire on the user's terminal to collect feedback information. The user answers the questionnaire and sends the results to the server.

[0083] Step 2:

[0084] Server: Analyzes the collected survey results and comments made during the tour, generates a feedback report, and provides it to the museum.

[0085] Step 3:

[0086] Server: Generates information recommending products from the museum store based on the exhibits and artists that the user is interested in during the tour, and sends this information to the user's device.

[0087] Step 4:

[0088] Device: Display related products on the user's device and provide a purchase link. For example, related products for "Famous Painting A" are displayed, and the user can purchase them via a link to an e-commerce site.

[0089] In this way, through the specific processing content at each step, users can have a high-quality experience through the virtual museum.

[0090] Example 1

[0091] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0092] There is a need for a system that allows general visitors to easily and effectively view cultural assets in museums. However, existing systems have issues such as insufficient digital archiving and the inability to communicate in real time between users and guides. Furthermore, they lack the ability to provide a personalized experience based on users' interests, which leads to a decline in user satisfaction.

[0093] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0094] In this invention, the server includes means for digitally archiving the museum's cultural assets, means for displaying the digitized cultural assets on an information processing terminal, and means for recommending the most appropriate tour guide based on the user's interests. This enables digital archiving of cultural assets and personalized online viewing based on the archived data. The server also includes means for extracting metadata from received digital data, converting it into a storage format, and storing it in a database; means for analyzing the user's viewing request and searching for tour guide information in the database; means for requesting the tour guide to confirm and approve the date and time of the viewing; means for sending a tour start notification and preloading the necessary digital data; and means for receiving user operations and questions during the viewing of the cultural assets, converting them into text, translating them into multiple languages, and sending them to the tour guide. This enables real-time two-way communication between the user and the tour guide, providing a richer online viewing experience.

[0095] An "art museum" is a facility that collects, preserves, exhibits, and makes available to the public cultural assets and works of art.

[0096] "Cultural properties" refer to items, documents, and exhibits that have historical, cultural, or academic value.

[0097] "Digital archiving" is the process of preserving physical cultural assets as digital data to increase their convenience.

[0098] "Information processing terminal" refers to a device such as a computer, tablet, or smartphone that displays and manipulates digital data.

[0099] "User" means an individual or group who uses the system to view digitally archived cultural property.

[0100] A "facilitator" is an expert or guide who provides explanations about cultural properties to users on the system.

[0101] "Online viewing" is the experience of viewing digitally archived cultural assets via the Internet.

[0102] "Two-way communication" is a function that allows users and moderators to exchange messages and information in real time.

[0103] "Metadata" is additional information about digital data, such as the artist's name, the year of production, and the technique used.

[0104] A "database" is a system for efficiently storing, managing, and searching multiple digital data.

[0105] "Multilingual translation" is a function that translates into different languages ​​and supports communication between users and facilitators who speak different languages.

[0106] The present invention is a system that digitally archives cultural assets in art museums and provides a virtual museum that users can access online. The system of the present invention operates primarily on a server and functions in conjunction with information processing terminals (user terminals and guide terminals). Specific embodiments of each function are described below.

[0107] server

[0108] The server receives and temporarily stores digital data (images, videos, text, etc.) of cultural assets provided by the museum. At this time, the server analyzes the digital data and uses image processing algorithms and natural language processing technology to extract metadata (artist name, year of production, technique, etc.). The server then converts the digital data into an appropriate storage format and stores it in a dedicated database. The server also indexes the data, making it quickly searchable.

[0109] For example, if a museum provides a high-resolution image of "Famous Painting A," the server stores the image in a temporary directory and uses image processing algorithms to extract the artist's name and the year of production. The server then converts the image along with the extracted metadata into an appropriate format and stores it in the database.

[0110] user

[0111] A user accesses the virtual museum from their device and requests a tour guide. For example, if the user inputs their desire to "learn about Impressionism," this information is sent to the server. The server analyzes this request and searches for guide information in its database. The search results list guides who are knowledgeable about Impressionism and provides this list to the user. The user selects the desired guide and requests confirmation and approval of the tour date and time from the server.

[0112] Specifically, when a user requests a "tour guide of contemporary art," the server searches the database for guides knowledgeable about contemporary art and presents a list to the user. The user selects the desired guide and sends the information to the server.

[0113] Guide terminal

[0114] When the tour begins, the server sends a notification to the user's and guide's devices. The server also pre-loads the digital data necessary for the tour. The guide uses a dedicated application to operate the exhibits while providing commentary on the cultural properties. The content of this commentary and screen operation information are sent to the user in real time via the server.

[0115] For example, while the guide is explaining "Famous Painting A," the user may ask a question via chat, such as, "I'd like to know about the techniques used in this painting." The question is sent to the guide via the server and translated as necessary. The guide can then answer the question in real time.

[0116] Server and user terminal

[0117] After the tour ends, the server displays a questionnaire on the user's device and collects responses. The server analyzes the collected questionnaire results and comments made during the tour to generate a feedback report and provide it to the museum, providing specific data for service improvement. Furthermore, based on the cultural assets the user showed interest in during the tour, the server recommends related museum store products to the user's device and provides a link to purchase them.

[0118] For example, if a user is interested in a replica of "Famous Painting A," the system will display related products and provide a link to purchase them, enabling museums to secure new revenue streams and providing visitors with a high-quality, guided museum experience from a remote location.

[0119] Example prompts for generative AI models

[0120] "Please detail the steps a user can take to request an Impressionist tour guide in your virtual museum system."

[0121] "Please explain in detail how the server receives and stores data from the museum in the digital archive system."

[0122] "Please detail the steps your server takes to display and collect a survey from users after the tour has finished."

[0123] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0124] Step 1: Receiving and temporarily storing digital data

[0125] The server receives digital data of cultural assets (images, videos, text, etc.) from the museum and stores the received data in a temporary directory.

[0126] Input: Digital data provided by the museum

[0127] Output: Digital data stored in a temporary directory

[0128] Specifically, when the museum uploads a high-resolution image file of "Masterpiece A," the server temporarily stores this data in a storage directory.

[0129] Step 2: Extracting Metadata

[0130] The server analyzes the digital data stored in the temporary storage directory and extracts metadata, such as the artist's name, production year, and technique, using image processing algorithms and natural language processing technology.

[0131] Input: Digital data stored in a temporary directory

[0132] Output: Extracted metadata

[0133] Specifically, the server automatically recognizes the artist's name, year of production, and techniques used from the image of "Masterpiece A" and extracts them as metadata.

[0134] Step 3: Convert and save the data format

[0135] The server converts the digital data along with the extracted metadata into a designated storage format, and stores the converted data in a dedicated database.

[0136] Input: Digital data, extracted metadata

[0137] Output: Data converted to a suitable storage format, data stored in a database

[0138] Specifically, it converts large image files into a format suitable for web display and stores the converted data and metadata in a database.

[0139] Step 4: Indexing

[0140] The server indexes the stored data, which allows for fast searches.

[0141] Input: Data stored in a database

[0142] Output: Indexed database

[0143] Specifically, the server adds metadata such as the artist's name and technique to the index, making it easy to search later.

[0144] Step 5: Request a tour guide

[0145] Users access the virtual museum from their own devices and request the tour guide of their choice, for example, "I want to learn about Impressionism," and send the request to the server.

[0146] Input: User's tour guide request

[0147] Output: Request sent to the server

[0148] Specifically, the user requests a "contemporary art tour guide" and the server receives the request, "I would like to learn about Impressionism."

[0149] Step 6: Request analysis and guided search

[0150] The server analyzes the user's request, searches for the moderator's information in the database, and lists detailed guides to "Impressionism" and provides them to the user.

[0151] Input: User request

[0152] Output: Guide list as search results

[0153] Specifically, the server searches a database for detailed guides on "Impressionism," lists them, and provides them to the user.

[0154] Step 7: View and select guide list

[0155] The user browses the list of guides sent from the server, selects the guide he or she wants, and sends the selected information to the server.

[0156] Input: Guide list from the server

[0157] Output: Selected guide information

[0158] Specifically, the user selects the desired guide from the displayed guides and sends that information to the server.

[0159] Step 8: Review and approve your guide

[0160] The server requests the selected guide to confirm and approve the viewing date and time. If the guide approves, the information is sent to the user to confirm the viewing details.

[0161] Input: User-selected guide information

[0162] Output: Guide approval status, notification to user

[0163] Specifically, the server requests the selected guide to confirm the viewing date and time, and notifies the user after approval.

[0164] Step 9: Notification of viewing start and data loading

[0165] When the viewing start time arrives, the server sends a notification to the user and guide's devices, and also preloads the digital data necessary for the viewing.

[0166] Input: Viewing start time

[0167] Output: Viewing start notification, loaded digital data

[0168] Specifically, the server sends a notification that viewing has begun and loads exhibit information.

[0169] Step 10: Guide operation and explanation

[0170] The guide terminal uses a dedicated application to operate the exhibits while providing commentary, and transmits the commentary and screen operation information to the user in real time.

[0171] Input: Exhibit operated by the guide

[0172] Output: Real-time commentary and control information

[0173] Specifically, the guide operates "Masterpiece A" while providing commentary, and transmits this information to the user in real time.

[0174] Step 11: User Actions and Questions

[0175] The user device allows them to view the virtual space in detail and send questions using the chat function.

[0176] Input: User question

[0177] Output: The question sent to the server

[0178] Specifically, when a user submits a question such as "What is this technique?", the question is sent to the server.

[0179] Step 12: Receiving and answering questions

[0180] The server converts user questions into text, translates them if necessary, and sends them to the guide, who then answers the questions in real time.

[0181] Input: User question

[0182] Output: The textual questions sent to the guide

[0183] Specifically, the server receives the user's question, translates it, and sends it to the guide, who then answers it.

[0184] Step 13: Display the survey and collect responses

[0185] After viewing, the server displays a questionnaire on the user's device and collects responses.

[0186] Input: Survey display request after viewing ends

[0187] Output: Collected survey responses

[0188] Specifically, the server displays a questionnaire on the user's device asking, "What did you think of the tour?" and collects the user's responses.

[0189] Step 14: Analyze and provide feedback

[0190] The server analyzes the collected survey results and comments made during the tour, generates a feedback report, and provides it to the museum.

[0191] Input: Collected survey responses and comments made during the tour

[0192] Output: Feedback report

[0193] Specifically, the server generates a feedback report based on the analysis results and provides it to the museum.

[0194] Step 15: Recommend a product and provide a link to purchase

[0195] The server recommends related museum store products based on the exhibits and artists the user has shown interest in during the tour and provides a purchase link to the user's device.

[0196] Input: Exhibits and artists that the user is interested in

[0197] Output: Recommended products and purchase links

[0198] Specifically, if a user is interested in a replica of "Famous Painting A," related products will be displayed and a purchase link will be provided.

[0199] (Application example 1)

[0200] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0201] In traditional art museums and stores, due to physical distance and time constraints, visitors had to travel to the actual site to view and purchase artworks in detail. Furthermore, due to a lack of guidance tailored to the individual interests of visitors, it was difficult to provide a satisfying experience.

[0202] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0203] In this invention, the server includes means for archiving digital data of artworks, means for displaying the digitized artworks on a terminal device, means for recommending the most suitable guide based on the user's interests, means for conducting an online tour in which the guide explains the artworks, means for supporting real-time two-way communication between the guide and the user, means for archiving digital data of store products, means for recommending the most suitable product guide based on the user's interests, and means for conducting an online tour in which the guide explains and suggests details about the products. This allows visitors to view and purchase artworks and products in detail without being restricted by physical distance or time. Furthermore, by providing guidance tailored to the user's individual interests, a highly satisfying experience can be achieved.

[0204] "Artworks" refers to artworks such as paintings, sculptures, and crafts that are exhibited in museums and galleries.

[0205] "Digital data" refers to information such as images, videos, and text recorded in electronic form.

[0206] "Archiving" refers to the process of storing information and data for a long period of time, organizing and storing it so that it can be easily searched and used later.

[0207] "End-device" refers to a terminal device that is directly operated by the user, and includes smartphones, tablets, PCs, smart glasses, etc.

[0208] A "guide" is someone with the expertise to provide users with detailed information about artworks and products and conduct online tours.

[0209] "Recommendation" means selecting the most suitable guide or product based on the user's interests and proposing it to the user.

[0210] An "online tour" is a virtual tour that takes place in real time over the Internet.

[0211] "Two-way communication" refers to a form of communication in which both parties (guide and user) can exchange information with each other.

[0212] "Products" refers to all items sold in stores, including fashion items, home appliances, stationery, etc.

[0213] "Details" are detailed descriptions or information about a particular subject.

[0214] "Proposal" means introducing solutions or products that meet the user's needs.

[0215] "Store" refers to a physical location or online platform for offering goods or services.

[0216] This invention is a system that digitally archives artworks and store merchandise and provides online guides and tours to users. Specific embodiments of each function will be described below.

[0217] 1. Archiving digital data

[0218] The server receives and temporarily stores product data (images, videos, text, etc.) provided by art objects or stores. When museums or stores upload high-resolution images of specific art objects or products, the server receives this data and stores it in a temporary storage directory. The server then analyzes the digital data and extracts metadata (e.g., artist name, year of creation, technique). This data is converted into an appropriate format and stored in a dedicated database. Indexing allows for quicker searching and filtering later.

[0219] 2. Recommendations based on user interests

[0220] Users access the system from their own devices (smartphones, tablets, smart glasses, etc.) and request guidance on a specific category, product, or artwork. For example, if a user enters a request such as "I'd like to see the latest winter fashion items," this information is sent to the server. The server analyzes this request and searches for guide information in its database. It then creates a list of the most suitable guides and sends this list to the user. The user then selects the desired guide from the displayed list and sends the information to the server.

[0221] 3. Online tours

[0222] When the tour begins, the server sends a notification to the user and guide's devices that the tour is about to begin, and pre-loads the digital data necessary for the tour. The guide uses a dedicated application to operate the products and artworks while providing commentary. At this time, the guide's device sends the commentary and screen operation information to the user in real time via the server. Users can operate the virtual space on their own devices to view the artworks and products in detail. Users can also send questions using the chat function. The server receives questions from users, converts them to text, and sends them to the guide. These questions are translated as needed, making multilingual support possible. The guide answers the questions received from users in real time.

[0223] 4. Secondary Features of the Service

[0224] After the tour ends, the server displays a questionnaire on the user's device and collects their responses. The collected questionnaire results and comments made during the tour are analyzed, and a feedback report is generated and provided to the museum or store. This allows for service improvements. Furthermore, based on the art pieces and products that the user showed interest in during the tour, related products are recommended and a purchasing link is provided to the user's device. For example, if a user is interested in the latest winter fashion items, related products are displayed and a link to purchase them is provided. This provides museums and stores with a new source of revenue and allows users to enjoy a high-quality guided experience from a remote location.

[0225] Adding specific examples

[0226] For example, if a user requests to "see the latest winter fashion items," the server will recommend the most suitable guide, who will then introduce "latest winter fashion item A" and explain its features and materials. Also, if a user asks a question about a product during the tour, the server will convert the question into text and transmit it to the guide in real time, providing an environment where an answer can be obtained immediately.

[0227] Prompt Sentence Examples

[0228] User request: "I want to see the latest winter fashion items"

[0229] Guide's answer: "Here's the latest winter fashion item, item A. It's made by a famous designer and is made from high-quality cashmere and wool. Its unique style and comfort make it a popular item in 2023."

[0230] This invention allows users to enjoy the experience of appreciating and purchasing high-quality artworks and merchandise without being physically restricted, and also provides customized guidance based on the user's individual interests, resulting in a highly satisfying service.

[0231] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0232] Step 1:

[0233] The server receives product data (images, videos, text, etc.) provided by the art works or stores and stores them in a temporary directory. The input is high-resolution images and detailed information provided by the museum or store, and the output is the temporarily stored data.

[0234] Step 2:

[0235] The server analyzes the digital data stored in the temporary storage directory and extracts metadata (artist name, production year, technique, etc.). The input is the stored digital data, and the output is the extracted metadata. Specific operations include using image recognition software and text analysis algorithms.

[0236] Step 3:

[0237] The server converts the analyzed digital data into an appropriate format and stores it in a database. The input is metadata and digital data, and the output is digital data stored in the database. Specifically, it performs data format conversion and indexing.

[0238] Step 4:

[0239] A user accesses the system from a terminal and requests information about a specific category, product, or artwork. The input is the user's request, and the output is the request information being sent to the server. Specific operations include receiving input through the user interface.

[0240] Step 5:

[0241] The server analyzes the user's request and searches for guide information in the database. The input is the user's request information, and the output is a list of the most suitable guides. The specific operation uses a search algorithm based on the request content.

[0242] Step 6:

[0243] The server sends a list of the most suitable guides to the user. The input is the list of guides, and the output is the guide information displayed on the user's terminal. Specific operations include data format conversion and transmission.

[0244] Step 7:

[0245] The user selects the desired guide from the list and sends the information to the server. The input is the selected guide information, and the output is sending the selected information to the server. The specific operation involves operating the selection interface.

[0246] Step 8:

[0247] When the tour start time arrives, the server sends a notification of the tour start to the user and guide's terminals and loads the necessary digital data in advance. The input is the tour start time and digital data, and the output is the notification and the loaded data on the terminal. Specific operations include schedule management and data transmission.

[0248] Step 9:

[0249] The guide uses a dedicated application to operate the products and artworks while providing commentary. The input is the loaded digital data and the guide's commentary, and the output is real-time commentary and screen operation information. Specific operations include voice input and screen operation.

[0250] Step 10:

[0251] Users operate the virtual space on their devices to view artworks and products in detail. They also use the chat function to send questions. The input is the user's operations and questions, and the output is the displayed virtual space and the questions sent. Specific operations include operating the user interface and sending questions.

[0252] Step 11:

[0253] The server receives the user's question, converts it into text, and sends it to the guide. The input is the user's question, and the output is the converted text and the transmitted question. Specific operations include voice recognition software and data transmission.

[0254] Step 12:

[0255] After the tour ends, the server displays a questionnaire on the user's device and collects responses. The input is the questionnaire form and the user's responses, and the output is the collected questionnaire data. Specifically, the server displays the questionnaire form and collects responses.

[0256] Step 13:

[0257] The server analyzes the collected survey results and comments made during the tour, generates a feedback report, and provides it to museums and stores. The input is the survey results and comments, and the output is the generated feedback report. Specific operations include data analysis and report generation.

[0258] Step 14:

[0259] The server recommends related products and provides purchase links based on the artworks and products that the user shows interest in during the tour. The input is the user's interest information and related product data, and the output is the recommended products and purchase links. Specific operations include data mining and link generation.

[0260] In this way, users can enjoy the experience of viewing and purchasing high-quality artworks and merchandise without being restricted by physical constraints, and by providing customized guidance based on individual interests, a highly satisfying service is realized.

[0261] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0262] The present invention is a system that digitally archives museum exhibits and provides a virtual museum that users can access online. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, a more personalized experience is provided. Specific embodiments of each function are described below.

[0263] 1. Digital archiving of museums

[0264] server

[0265] The server receives digital data (images, videos, text, etc.) of exhibits provided by the museum and temporarily stores it. For example, when the museum uploads a high-resolution image of "Famous Painting A," the server receives this data and moves it to a temporary storage directory. The server then analyzes the digital data and extracts metadata (e.g., artist name, year of production, technique used). The server then converts the data into an appropriate format and stores it in a dedicated database. The server also indexes the data to enable quick searches and filtering later.

[0266] 2. Online tour guide matching

[0267] user

[0268] A user accesses the virtual museum from their device and requests a tour guide. For example, if they input their desire to "learn about Impressionism," this information is sent to the server. The server analyzes this request and searches for guide information in its database. For example, it creates a list of guides who are knowledgeable about "Impressionism" and sends this list to the user. The user selects the desired guide from the displayed guides and sends that information to the server. The server then requests confirmation and approval of the tour date and time from the selected guide, and if approval is granted, sends the confirmed information to the user.

[0269] 3. Conducting Virtual Tours

[0270] Server, user terminal, guide terminal

[0271] When the tour begins, the server sends a notification to the user and guide's devices that the tour is about to begin, and pre-loads the digital data necessary for the tour. The guide uses a dedicated application to operate the exhibits while giving their explanation. At this time, the guide's device sends the explanation and screen operation information to the user in real time via the server. Users can operate the virtual space on their own devices and view the exhibits in detail. Users can also send questions using the chat function. The server receives questions from users, converts them to text, and sends them to the guide. These questions are translated as needed, making multilingual support possible. The guide answers the users' questions in real time.

[0272] 4. Secondary Features of the Service

[0273] Server, user terminal

[0274] After the tour ends, the server displays a questionnaire on the user's device and collects feedback information. The collected questionnaire results and comments made during the tour are analyzed, and a feedback report is generated and provided to the museum. Based on the exhibits and artists that the user showed interest in during the tour, the server also generates information recommending products from the museum store and sends this to the user's device. Related products are displayed on the user's device, and a link to purchase them is provided. For example, related products for "Famous Painting A" are displayed, and the user can purchase them via a link to an e-commerce site.

[0275] 5. Incorporating an Emotional Engine

[0276] Emotion Engine

[0277] The emotion engine analyzes the user's facial expression data, tone of voice, and text input data collected in real time from the user's device to recognize the user's emotional state, for example, determining whether the user is excited or bored by a particular exhibit.

[0278] Use of Emotional Data

[0279] Server, guide terminal

[0280] The emotion engine analyzes the user's emotional state and sends it back to the guide terminal in real time via the server. Based on this information, the guide can dynamically adjust the content of the commentary or provide additional information to pique the user's interest. For example, if the user is not interested in a particular exhibit, the guide can introduce other related exhibits.

[0281] Feedback Improvements

[0282] server

[0283] The server generates feedback to improve the user experience based on the emotional data analyzed by the emotion engine and provides it to the museum, which can then use it to improve the exhibits and tour content.

[0284] In this way, the system of the present invention recognizes and reflects user emotions in real time, providing a more personalized and high-quality online museum experience.

[0285] The processing flow will be explained below.

[0286] 1. Digital archiving of museums

[0287] Step 1:

[0288] Server: Provides a dedicated form on the website for museum staff to upload digital data (images, videos, text, etc.) of exhibits.

[0289] Step 2:

[0290] Server: Receives uploaded digital data and moves it to a temporary storage directory. For example, if a museum uploads a high-resolution image of "Famous Painting A," the data is stored in that directory.

[0291] Step 3:

[0292] Server: Analyzes the received digital data and executes analysis scripts to extract metadata (artist name, year of production, technique, etc.). The extracted metadata is stored in a database along with the data.

[0293] Step 4:

[0294] Server: Indexes the data to enable quick searches and filtering later. The indexed data is efficiently managed in a database.

[0295] 2. Online tour guide matching

[0296] Step 1:

[0297] User: The user accesses the virtual museum website from their device, enters the tour guide request screen, and enters "I would like a tour of the Impressionists." This information is sent to the server.

[0298] Step 2:

[0299] Server: Analyzes the user's request, searches for guide information in the database, generates a list of target guides, and sends it to the user's device.

[0300] Step 3:

[0301] User: The user selects the desired guide from the displayed guides and sends the selected information to the server. For example, if the user selects "Guide B," the information is sent to the server.

[0302] Step 4:

[0303] Server: Requests confirmation and approval of the tour date and time from the selected guide, and sends the confirmation information to the user once approval is received.

[0304] 3. Conducting Virtual Tours

[0305] Step 1:

[0306] Server: When the tour start time arrives, the server sends a tour start notification to the user and guide's devices and preloads the digital data required for the tour.

[0307] Step 2:

[0308] Guide: The guide uses a dedicated application to operate the exhibits and provide commentary. Screen operations and commentary are distributed to users in real time via a server.

[0309] Step 3:

[0310] Users: Users can operate the virtual space on their own devices, view the exhibits in detail, and send questions to the guide using the chat function.

[0311] Step 4:

[0312] Server: Converts user questions into text and sends it to the guide. If necessary, translates the questions to support multiple languages.

[0313] Step 5:

[0314] Guide: The guide answers users' questions in real time.

[0315] 4. Secondary Features of the Service

[0316] Step 1:

[0317] Terminal: After the tour ends, the server displays a questionnaire on the user's terminal to collect feedback information. The user answers the questionnaire and sends the results to the server.

[0318] Step 2:

[0319] Server: Analyzes the collected survey results and comments made during the tour, generates a feedback report, and provides it to the museum.

[0320] Step 3:

[0321] Server: Generates information recommending products from the museum store based on the exhibits and artists that the user is interested in during the tour, and sends this information to the user's device.

[0322] Step 4:

[0323] Device: Display related products on the user's device and provide a purchase link. For example, related products for "Famous Painting A" are displayed, and the user can purchase them via a link to an e-commerce site.

[0324] 5. Incorporating an Emotional Engine

[0325] Step 1:

[0326] Device: Obtains facial expression data, tone of voice, text input, etc. in real time from the user's device.

[0327] Step 2:

[0328] Server: Sends the acquired data to the emotion engine to analyze the user's emotional state, for example, analyzing the emotions (excitement, indifference, etc.) when the user is viewing a particular exhibit.

[0329] Step 3:

[0330] Server: Based on the analysis results, the user's emotional state is fed back to the guide device in real time.

[0331] Step 4:

[0332] Guide: The guide dynamically adjusts its commentary based on the emotional feedback it receives, for example, providing more detailed information about an exhibit that excites the user.

[0333] Step 5:

[0334] Server: After the tour ends, the emotion engine generates feedback based on the emotional data analyzed and provides it to the museum.

[0335] In this way, the specific processing content at each step ensures that users can have a high-quality experience through the virtual museum. In addition, by incorporating an emotion engine, a more personalized and engaging experience is provided.

[0336] Example 2

[0337] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0338] Conventional digital archive systems for museum exhibits simply store and display digital data of exhibits, limiting the user experience. They also lack digital tour guide recommendations and two-way communication functions, making it difficult to provide a personalized experience tailored to participants. Furthermore, they lack the ability to recognize users' emotions in real time and reflect them in the guide's commentary, making it difficult to respond flexibly to users' interests and level of understanding.

[0339] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for digitally archiving museum exhibits, means for displaying the digitized exhibits on a user terminal, means for recommending an optimal tour guide based on the user's interests and concerns, means for conducting an online tour in which the guide explains the exhibits, means for supporting two-way communication between the guide and the user in real time, means for analyzing the user's emotional data and providing feedback to the guide in real time, and means for collecting feedback from the user after the tour ends and providing the feedback to the museum. This allows the user to experience a high-quality virtual tour of the museum's exhibits through the digital archive, and receive a personalized explanation based on the real-time emotional data.

[0340] An "art museum" is a facility that displays exhibits for the general public and aims to promote education and culture.

[0341] "Exhibits" refers to cultural assets and artworks such as paintings, sculptures, crafts, and photographs displayed in art museums.

[0342] "Digital archiving" is the process of recording, preserving, and managing exhibits as digital data (images, videos, text, etc.).

[0343] "User Terminal" refers to a device, such as a computer, smartphone, or tablet, that a User uses to access the System via the Internet.

[0344] A "tour guide" is a professional or function that provides commentary and explanations about museum exhibits, either online or in person.

[0345] The "recommendation method" is part of a system that uses algorithms and filtering functions to select and suggest the most suitable tour guide based on the user's interests.

[0346] An "online tour" is a service that allows users to virtually view museum exhibits using the Internet while receiving real-time commentary from a guide.

[0347] "Two-way communication" refers to the interactive function that allows users and guides to exchange information in real time.

[0348] "Emotional data" refers to information about a user's psychological state and emotions analyzed from facial expressions, tone of voice, text input, etc.

[0349] "Feedback" refers to information such as opinions, impressions, and evaluations collected from users after the tour, and is used to improve and evaluate the service.

[0350] The present invention is a system that digitally archives museum exhibits and provides a virtual museum that users can access online. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system provides a more personalized experience. Specific embodiments of each function are described below.

[0351] Digital archiving of museum exhibits

[0352] server

[0353] The server receives digital data (images, videos, text, etc.) of exhibits provided by the museum and temporarily stores it. For example, if the museum uploads a high-resolution image of "Famous Painting A," the server receives this data and moves it to a temporary storage directory. The server analyzes the received digital data and extracts metadata (artist name, year of production, technique, etc.). Software such as image analysis algorithms is used for the analysis. The analyzed data is converted into an appropriate format and stored in a dedicated database. In addition, indexing the data allows for faster searching and filtering later.

[0354] Online tour guide matching

[0355] user

[0356] A user accesses the virtual museum from their device and requests a tour guide. For example, if they enter their request as "I would like to learn about Impressionism," this information is sent to the server. The server analyzes the user's request and searches for guide information in its database. For example, it creates a list of guides who are knowledgeable about "Impressionism" and sends this list to the user. The user selects the desired guide from the displayed guides and sends that information to the server. The server then requests confirmation and approval of the tour date and time from the selected guide, and if approval is granted, sends the confirmed information to the user.

[0357] Conducting a virtual tour

[0358] Server, user terminal, guide terminal

[0359] When the tour begins, the server sends a notification to the user's and guide's devices and pre-loads the digital data necessary for the tour. The guide uses a dedicated application to operate the exhibits while providing commentary. For example, the guide might display details of "Masterpiece A" on the screen while explaining its background and techniques. The guide's device sends the commentary and screen operation information to the user in real time via the server. Users can operate the virtual space on their own devices and view the exhibits in detail. Users can also send questions using the chat function. The server receives questions from users, converts them to text, and sends them to the guide. These questions are translated as needed, making multilingual support possible. The guide then answers the user's questions in real time.

[0360] Service Sub-Features

[0361] Server, user terminal

[0362] After the tour ends, the server displays a questionnaire on the user's device to collect feedback information. For example, feedback is collected by having the user answer the question, "How satisfied were you with this tour?" The collected feedback information is analyzed and provided to the museum. The server analyzes the collected feedback and the user's interests, generates information recommending related museum store products, and sends this information to the user's device. For example, related products for "Famous Painting A" are displayed, and the user can purchase them via a link to an e-commerce site.

[0363] Incorporating an emotion engine

[0364] Emotion engine, server

[0365] The emotion engine analyzes the user's facial expression data, tone of voice, and text input content obtained in real time from the user's device to recognize the user's emotional state. For example, it determines whether the user is excited or bored by a particular exhibit. This information is fed back to the guide device via the server. The guide then dynamically adjusts the content of the commentary based on this information. For example, if the user shows no interest in a particular exhibit, the guide will introduce other related exhibits. The server generates feedback to improve the user experience based on the emotional data analyzed by the emotion engine and provides it to the museum.

[0366] Examples of concrete examples and prompts

[0367] For example, if an art museum were to digitally archive an "Impressionist Exhibition," it would follow the steps below.

[0368] 1. The museum uploads a high-resolution image of "Masterpiece A" to the server.

[0369] 2. The server analyzes the image data and extracts metadata such as the artist's name and technique.

[0370] 3. The user requests, "I want to learn about Impressionism," and the server matches them with a suitable tour guide.

[0371] 4. When the tour start time arrives, the server notifies the user and guide devices and the virtual tour begins.

[0372] 5. The guide explains the details of the exhibits, and when the user asks questions, they are sent to the guide via the server.

[0373] 6. After the tour, a survey will be displayed on the user's device to collect feedback information.

[0374] Prompt Sentence Examples

[0375] I want to book a virtual tour about Impressionism.

[0376] I want to know more about Masterpiece A.

[0377] The above is an embodiment of the present invention, which allows users to experience a high-quality virtual tour and receive personalized commentary in real time.

[0378] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0379] Step 1:

[0380] The server receives the digital data from the museum.

[0381] Input: High-resolution images, videos, and text data uploaded by museums.

[0382] Data processing: The input data is saved in a temporary directory.

[0383] Output: Digital data saved in temporary directory.

[0384] Specific operation: When the museum uploads a high-resolution image of "Famous Painting A," the server receives this data and moves it to a temporary storage directory.

[0385] Step 2:

[0386] The server analyzes the received digital data and extracts the metadata.

[0387] Input: Digital data stored in the temporary directory.

[0388] Data processing: Analyze the digital data using software such as image analysis algorithms.

[0389] Output: Extracted metadata (artist name, year of production, technique, etc.).

[0390] Specific operation: The server analyzes the stored image data and automatically extracts information about the artist and era of the work.

[0391] Step 3:

[0392] The server converts the parsed data into an appropriate format and stores it in a database.

[0393] Input: Extracted metadata and original digital data.

[0394] Data processing: Image data is converted into a specific compressed format and stored in a database along with metadata.

[0395] Output: Digital data and metadata stored in a database.

[0396] Specific operation: The server converts the analyzed data into an appropriate format and stores the digital data and metadata in a database.

[0397] Step 4:

[0398] Users access the virtual museum from their own devices and request a tour guide.

[0399] Input: A tour guide request sent from the user's device (e.g., "I want to learn about Impressionism").

[0400] Data processing: The server receives and analyzes the user's request.

[0401] Output: The tour guide request sent to the server.

[0402] What it does: A user types in "I want to learn about Impressionism," and this information is sent to the server.

[0403] Step 5:

[0404] The server analyzes the tour guide request and matches the most suitable guide.

[0405] Input: User request data.

[0406] Data processing: The server searches the guide information in the database and lists related guides.

[0407] Output: A list of guides that will be sent to the user.

[0408] What it does: The server compiles a list of detailed guides to "Impressionism" from its database and sends this list to the user.

[0409] Step 6:

[0410] The user selects the desired guide and submits it to the server, which requests confirmation and approval of the tour date and time.

[0411] Input: User selected guide information.

[0412] Data processing: The server confirms the tour date and time with the selected guide and requests approval.

[0413] Output: Tour information approved by the guide.

[0414] Specific operation: The user selects the desired guide and sends the information to the server. The server requests the guide to confirm and approve the tour date and time, and once approval is received, the confirmation information is sent to the user.

[0415] Step 7:

[0416] At the start time of the tour, the server sends a notification and pre-loads the digital data required for the tour.

[0417] Input: Confirmed tour information.

[0418] Data processing: Preload the necessary digital data.

[0419] Output: Tour start notification to user and guide devices.

[0420] Specific operation: When the tour start time arrives, the server sends a notification and loads the digital data required for the tour into the user's and guide's devices in advance.

[0421] Step 8:

[0422] The guide will provide commentary while operating the exhibits, and users can send questions via their terminal.

[0423] Input: Data about the exhibits operated by the guide, user question data.

[0424] Data processing: The server sends the commentary and screen operation information to the user, and sends the user's questions to the guide.

[0425] Output: Instructions sent to the user, user questions sent to the guide.

[0426] Specific operation: The guide explains "Famous Painting A," and when the user asks a question, the server sends the question to the guide, who then answers in real time.

[0427] Step 9:

[0428] The emotion engine analyzes the user's emotional data and provides feedback to the guide via the server.

[0429] Input: User facial expression data, tone of voice, and text input.

[0430] Data processing: The emotion engine analyzes the emotion data and provides feedback to the guide via the server.

[0431] Output: The user's emotional state, which is fed back to the guide.

[0432] Specific operation: The emotion engine analyzes the user's emotions and provides the guide with the user's excitement level or boredom level in real time.

[0433] Step 10:

[0434] After the tour ends, the server displays a questionnaire on the user's device to collect feedback information.

[0435] Input: User survey data after the tour.

[0436] Data processing: Analyze the collected survey results and generate a feedback report to provide to the museum.

[0437] Output: Feedback report provided to the museum, related product recommendations displayed on the user's device.

[0438] Specific operation: After the tour ends, a questionnaire is displayed on the user's device, and feedback information is collected and analyzed. The analysis results are provided to the museum, and related product recommendations are displayed on the user's device.

[0439] (Application example 2)

[0440] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0441] Conventional online virtual tour systems for museums and stores have struggled to provide a personalized experience based on the user's interests and emotions. Furthermore, they lacked the functionality to analyze the user's facial expressions and tone of voice to provide real-time product recommendations and information. This resulted in low user satisfaction.

[0442] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for digitalizing exhibits, means for displaying the digitalized exhibits on a user terminal, means for recommending an optimal guide based on the user's interests and concerns, means for conducting an online guided tour in which the guide explains the exhibits, means for supporting two-way communication between the guide and the user in real time, means for analyzing the user's facial expression data and voice data in real time to recognize the user's emotional state, and means for recommending products that will interest the user and providing additional information based on the user's emotional state. This enables a personalized and high-quality online experience that is tailored to the user's emotions and interests.

[0443] "Means of digitalizing exhibits" refers to the technology used to convert physical exhibits into digital data and store it in a database.

[0444] "Means for displaying digitalized exhibits on user terminals" refers to technology for displaying exhibits stored as digital data on terminals used by users.

[0445] "Means for recommending the most suitable guide based on the user's interests and concerns" refers to technology that analyzes the interests and concerns expressed by the user and automatically selects and recommends the guide that best suits them.

[0446] "Means for conducting online guided tours in which a guide explains exhibits" refers to technology that enables a guide to provide explanations about digitized exhibits and share the content with users in real time.

[0447] "Means for supporting two-way communication between a guide and a user in real time" refers to technology that enables a guide and a user to have two-way communication in real time.

[0448] "Means for analyzing a user's facial expression data and voice data in real time to recognize their emotional state" refers to technology that uses a camera or microphone to capture the user's facial expressions and voice, analyzes that data, and recognizes the user's emotional state in real time.

[0449] The "means for recommending products and providing additional information that may interest a user based on the user's emotional state" is a technology that dynamically provides products and related information that may be of interest to a user based on the user's recognized emotional state.

[0450] This invention is a system that allows users to experience exhibits and products in a virtual space, and provides personalized information by recognizing the user's emotions in real time. This system is constructed mainly using a server, a user terminal, and a guide terminal, and specific embodiments are described below.

[0451] Hardware and Software Use

[0452] The server is installed on the cloud and performs temporary data storage, analysis, and database management. Smartphones, tablets, PCs, etc. are used as user and guide devices, and these devices must be equipped with a camera and microphone. The main software used is as follows:

[0453] Google Cloud Vision API: A service for emotion analysis that analyzes image data and recognizes the user's facial expressions.

[0454] Firebase: A database service that processes and stores data in real time, managing user sentiment data and feedback information.

[0455] WebRTC: A technology that enables real-time communication, used for video chat and data streaming between users and guides.

[0456] Data acquisition and analysis

[0457] The server receives and temporarily stores digital data (images, videos, text, etc.) of exhibits and products provided by museums and stores. When high-resolution image data and detailed information are uploaded, the server analyzes it, extracts appropriate metadata (e.g., product name, price, description), and stores it in a database.

[0458] User information analysis and matching

[0459] When a user visits the virtual store, the user's device captures facial expression data using a camera and voice data using a microphone. This data is sent in real time to the Google Cloud Vision API, which analyzes the user's emotional state. The analysis results are stored in Firebase and fed back to the guide via the server.

[0460] Online tours and guided tours

[0461] When the tour begins, the server sends a notification to the user's and guide's devices and pre-loads the digital data necessary for the tour. The guide uses a dedicated application to interact with the exhibits and products while providing commentary, which is then sent to the user's device in real time. The user can move their viewpoint in any direction within the virtual space to view the exhibits in detail. They can also use the chat function to send questions to the guide.

[0462] Feedback and recommendation system

[0463] After the tour ends, the server displays a questionnaire screen on the user's device to collect feedback. Based on this feedback information and emotional data obtained during the tour, the server generates recommendation information for products and exhibits that the user showed interest in and provides it to the user's device. This information also includes links to related products and services, allowing users to smoothly make purchases or check details.

[0464] Examples of concrete examples and prompts

[0465] As a concrete example, if a user is exploring the wearable devices section of a virtual store and their smartphone camera captures their facial expression and detects "excitement," the system can analyze this data and automatically display more information and reviews related to the product they are particularly interested in (e.g., the latest smartwatch).

[0466] Example prompt sentence:

[0467] "Analyze users' facial expressions and tone of voice in real time and make appropriate product recommendations based on their emotional state (e.g., joy, anger, sadness, happiness, etc.). Use Firebase and Google Cloud Vision API to store user emotion data and notify the shopping guide in real time."

[0468] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0469] Step 1:

[0470] The server receives and temporarily stores digital data (e.g., images, videos, text) of exhibits and products provided by museums and stores. It then analyzes this data to extract metadata (e.g., product name, price, description) and stores it in a database. The input is digital data, and the output is metadata.

[0471] Step 2:

[0472] The user device captures the user's facial expression data using a camera and voice data using a microphone. This data is sent in real time to the Google Cloud Vision API, which analyzes the user's emotional state. The input is the user's facial expression and voice data, and the output is the analysis result of the emotional state.

[0473] Step 3:

[0474] The server stores the emotional state data obtained as a result of the analysis in Firebase, and uses this data to generate recommendation data for products that will interest the user. The input is the emotional state analysis results, and the output is recommendation data.

[0475] Step 4:

[0476] When a user accesses a virtual store, the server recommends the most suitable guide based on the user's interests. Based on input data about interests obtained from the user's device, the server searches for guide information in the database and lists the most suitable guides. The input is data about the user's interests, and the output is a list of guides.

[0477] Step 5:

[0478] At the start of the tour, the server sends notifications to the user and guide's devices and preloads digital data about the virtual store. The user and guide start a real-time video chat using WebRTC, and the server relays the communication between them. The input is a connection request from both devices, and the output is the establishment of the connection.

[0479] Step 6:

[0480] During the tour, the guide explains the exhibits and products, and the content is sent to the user in real time via the server. Users can use the chat function to send questions, which the server converts into text and sends to the guide. The input is the guide's explanation data and the user's question data, and the output is the textual question data and explanation content.

[0481] Step 7:

[0482] After the tour ends, the server displays a questionnaire screen on the user's device to collect feedback. The collected feedback information and emotional data collected during the tour are analyzed, and recommendation information is generated for products and exhibits that the user showed interest in. The input is feedback data and emotional data, and the output is recommendation information.

[0483] Step 8:

[0484] The server sends the final recommendation information to the user's device and provides detailed information and a purchase link for related products and services, allowing the user to smoothly purchase products or check details. The input is the generated recommendation information, and the output is the detailed information and purchase link displayed on the user's device.

[0485] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0486] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0487] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0488] [Second embodiment]

[0489] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0490] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0491] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0492] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0493] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0494] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0495] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0496] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0497] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0498] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0499] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0500] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0501] The present invention provides a system that digitally archives art museum exhibits and provides a virtual museum that users can access online. Specific embodiments of each function are described below.

[0502] 1. Digital archiving of museums

[0503] server

[0504] The server receives digital data (images, videos, text, etc.) of exhibits provided by the museum and temporarily stores it. For example, when the museum uploads a high-resolution image of "Famous Painting A," the server receives this data and stores it in a temporary storage directory. The server then analyzes the digital data and extracts metadata (e.g., artist name, year of production, technique used). The server then converts the data into an appropriate format and stores it in a dedicated database. The server also indexes the data to enable quick searches and filtering later.

[0505] 2. Online tour guide matching

[0506] user

[0507] A user accesses the virtual museum from their device and requests a tour guide. For example, if they input their desire to "learn about Impressionism," this information is sent to the server. The server analyzes this request and searches for guide information in its database. For example, it creates a list of guides who are knowledgeable about "Impressionism" and sends this list to the user. The user selects the desired guide from the displayed guides and sends that information to the server. The server then requests confirmation and approval of the tour date and time from the selected guide, and if approval is granted, sends the confirmed information to the user.

[0508] 3. Conducting Virtual Tours

[0509] Server, user terminal, guide terminal

[0510] When the tour begins, the server sends a notification to the user and guide's devices that the tour is about to begin, and pre-loads the digital data necessary for the tour. The guide uses a dedicated application to operate the exhibits while giving their explanation. At this time, the guide's device sends the explanation and screen operation information to the user in real time via the server. Users can operate the virtual space on their own devices and view the exhibits in detail. Users can also send questions using the chat function. The server receives questions from users, converts them to text, and sends them to the guide. These questions are translated as needed, making multilingual support possible. The guide answers the users' questions in real time.

[0511] 4. Secondary Features of the Service

[0512] Server, user terminal

[0513] After the tour ends, the server displays a survey on the user's device and collects responses. The collected survey results and comments made during the tour are analyzed, and a feedback report is generated and provided to the museum. This allows the museum to improve its services. Furthermore, based on the exhibits and artists that the user showed interest in during the tour, the server recommends products from the museum store and provides a purchasing link to the user's device. For example, if a user is interested in a replica of "Masterpiece A," related products will be displayed and a link to purchase them will be provided.

[0514] This will provide museums with new revenue streams and allow visitors to have a high-quality guided museum experience remotely.

[0515] The processing flow will be explained below.

[0516] 1. Digital archiving of museums

[0517] Step 1:

[0518] Server: Provides a dedicated upload form to receive digital data (images, videos, text, etc.) of exhibits provided by the museum. When a museum staff member uploads a high-resolution image of "Masterpiece A," the server receives this data and moves it to a directory for temporary storage.

[0519] Step 2:

[0520] Server: Analyzes the received digital data and runs analysis scripts to automatically extract metadata (artist name, year of production, technique, etc.) Once the metadata is extracted, the server stores this data in a digital archive database.

[0521] Step 3:

[0522] Server: Indexes the data to enable quick searches and filtering later. The indexed data is efficiently managed in a database.

[0523] 2. Online tour guide matching

[0524] Step 1:

[0525] Device: The user accesses the virtual museum website from their device, enters the tour guide request screen, and enters "I would like a tour of the Impressionists." This information is sent to the server.

[0526] Step 2:

[0527] Server: Analyzes the user's request, searches for guide information in the database, generates a list of target guides, and sends it to the user's device.

[0528] Step 3:

[0529] Terminal: The user selects the desired guide from the displayed guides and sends the selected information to the server. For example, if "Guide B" is selected, the information is sent to the server.

[0530] Step 4:

[0531] Server: Requests confirmation and approval of the tour date and time from the selected guide, and sends the confirmation information to the user once approval is received.

[0532] 3. Conducting Virtual Tours

[0533] Step 1:

[0534] Server: When the tour start time arrives, the server sends a tour start notification to the user and guide's devices and preloads the digital data required for the tour.

[0535] Step 2:

[0536] Guide: The guide uses a dedicated application to operate the exhibits and provide commentary. Screen operations and commentary are distributed to users in real time via a server.

[0537] Step 3:

[0538] Users: Users can operate the virtual space on their own devices, view the exhibits in detail, and send questions to the guide using the chat function.

[0539] Step 4:

[0540] Server: Converts user questions into text and sends it to the guide. If necessary, translates the questions to support multiple languages.

[0541] Step 5:

[0542] Guide: The guide answers users' questions in real time.

[0543] 4. Secondary Features of the Service

[0544] Step 1:

[0545] Terminal: After the tour ends, the server displays a questionnaire on the user's terminal to collect feedback information. The user answers the questionnaire and sends the results to the server.

[0546] Step 2:

[0547] Server: Analyzes the collected survey results and comments made during the tour, generates a feedback report, and provides it to the museum.

[0548] Step 3:

[0549] Server: Generates information recommending products from the museum store based on the exhibits and artists that the user is interested in during the tour, and sends this information to the user's device.

[0550] Step 4:

[0551] Device: Display related products on the user's device and provide a purchase link. For example, related products for "Famous Painting A" are displayed, and the user can purchase them via a link to an e-commerce site.

[0552] In this way, through the specific processing content at each step, users can have a high-quality experience through the virtual museum.

[0553] Example 1

[0554] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0555] There is a need for a system that allows general visitors to easily and effectively view cultural assets in museums. However, existing systems have issues such as insufficient digital archiving and the inability to communicate in real time between users and guides. Furthermore, they lack the ability to provide a personalized experience based on users' interests, which leads to a decline in user satisfaction.

[0556] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0557] In this invention, the server includes means for digitally archiving the museum's cultural assets, means for displaying the digitized cultural assets on an information processing terminal, and means for recommending the most appropriate tour guide based on the user's interests. This enables digital archiving of cultural assets and personalized online viewing based on the archived data. The server also includes means for extracting metadata from received digital data, converting it into a storage format, and storing it in a database; means for analyzing the user's viewing request and searching for tour guide information in the database; means for requesting the tour guide to confirm and approve the date and time of the viewing; means for sending a tour start notification and preloading the necessary digital data; and means for receiving user operations and questions during the viewing of the cultural assets, converting them into text, translating them into multiple languages, and sending them to the tour guide. This enables real-time two-way communication between the user and the tour guide, providing a richer online viewing experience.

[0558] An "art museum" is a facility that collects, preserves, exhibits, and makes available to the public cultural assets and works of art.

[0559] "Cultural properties" refer to items, documents, and exhibits that have historical, cultural, or academic value.

[0560] "Digital archiving" is the process of preserving physical cultural assets as digital data to increase their convenience.

[0561] "Information processing terminal" refers to a device such as a computer, tablet, or smartphone that displays and manipulates digital data.

[0562] "User" means an individual or group who uses the system to view digitally archived cultural property.

[0563] A "facilitator" is an expert or guide who provides explanations about cultural properties to users on the system.

[0564] "Online viewing" is the experience of viewing digitally archived cultural assets via the Internet.

[0565] "Two-way communication" is a function that allows users and moderators to exchange messages and information in real time.

[0566] "Metadata" is additional information about digital data, such as the artist's name, the year of production, and the technique used.

[0567] A "database" is a system for efficiently storing, managing, and searching multiple digital data.

[0568] "Multilingual translation" is a function that translates into different languages ​​and supports communication between users and facilitators who speak different languages.

[0569] The present invention is a system that digitally archives cultural assets in art museums and provides a virtual museum that users can access online. The system of the present invention operates primarily on a server and functions in conjunction with information processing terminals (user terminals and guide terminals). Specific embodiments of each function are described below.

[0570] server

[0571] The server receives and temporarily stores digital data (images, videos, text, etc.) of cultural assets provided by the museum. At this time, the server analyzes the digital data and uses image processing algorithms and natural language processing technology to extract metadata (artist name, year of production, technique, etc.). The server then converts the digital data into an appropriate storage format and stores it in a dedicated database. The server also indexes the data, making it quickly searchable.

[0572] For example, if a museum provides a high-resolution image of "Famous Painting A," the server stores the image in a temporary directory and uses image processing algorithms to extract the artist's name and the year of production. The server then converts the image along with the extracted metadata into an appropriate format and stores it in the database.

[0573] user

[0574] A user accesses the virtual museum from their device and requests a tour guide. For example, if the user inputs their desire to "learn about Impressionism," this information is sent to the server. The server analyzes this request and searches for guide information in its database. The search results list guides who are knowledgeable about Impressionism and provides this list to the user. The user selects the desired guide and requests confirmation and approval of the tour date and time from the server.

[0575] Specifically, when a user requests a "tour guide of contemporary art," the server searches the database for guides knowledgeable about contemporary art and presents a list to the user. The user selects the desired guide and sends the information to the server.

[0576] Guide terminal

[0577] When the tour begins, the server sends a notification to the user's and guide's devices. The server also pre-loads the digital data necessary for the tour. The guide uses a dedicated application to operate the exhibits while providing commentary on the cultural properties. The content of this commentary and screen operation information are sent to the user in real time via the server.

[0578] For example, while the guide is explaining "Famous Painting A," the user may ask a question via chat, such as, "I'd like to know about the techniques used in this painting." The question is sent to the guide via the server and translated as necessary. The guide can then answer the question in real time.

[0579] Server and user terminal

[0580] After the tour ends, the server displays a questionnaire on the user's device and collects responses. The server analyzes the collected questionnaire results and comments made during the tour to generate a feedback report and provide it to the museum, providing specific data for service improvement. Furthermore, based on the cultural assets the user showed interest in during the tour, the server recommends related museum store products to the user's device and provides a link to purchase them.

[0581] For example, if a user is interested in a replica of "Famous Painting A," the system will display related products and provide a link to purchase them, enabling museums to secure new revenue streams and providing visitors with a high-quality, guided museum experience from a remote location.

[0582] Example prompts for generative AI models

[0583] "Please detail the steps a user can take to request an Impressionist tour guide in your virtual museum system."

[0584] "Please explain in detail how the server receives and stores data from the museum in the digital archive system."

[0585] "Please detail the steps your server takes to display and collect a survey from users after the tour has finished."

[0586] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0587] Step 1: Receiving and temporarily storing digital data

[0588] The server receives digital data of cultural assets (images, videos, text, etc.) from the museum and stores the received data in a temporary directory.

[0589] Input: Digital data provided by the museum

[0590] Output: Digital data stored in a temporary directory

[0591] Specifically, when the museum uploads a high-resolution image file of "Masterpiece A," the server temporarily stores this data in a storage directory.

[0592] Step 2: Extracting Metadata

[0593] The server analyzes the digital data stored in the temporary storage directory and extracts metadata, such as the artist's name, production year, and technique, using image processing algorithms and natural language processing technology.

[0594] Input: Digital data stored in a temporary directory

[0595] Output: Extracted metadata

[0596] Specifically, the server automatically recognizes the artist's name, year of production, and techniques used from the image of "Masterpiece A" and extracts them as metadata.

[0597] Step 3: Convert and save the data format

[0598] The server converts the digital data along with the extracted metadata into a designated storage format, and stores the converted data in a dedicated database.

[0599] Input: Digital data, extracted metadata

[0600] Output: Data converted to a suitable storage format, data stored in a database

[0601] Specifically, it converts large image files into a format suitable for web display and stores the converted data and metadata in a database.

[0602] Step 4: Indexing

[0603] The server indexes the stored data, which allows for fast searches.

[0604] Input: Data stored in a database

[0605] Output: Indexed database

[0606] Specifically, the server adds metadata such as the artist's name and technique to the index, making it easy to search later.

[0607] Step 5: Request a tour guide

[0608] Users access the virtual museum from their own devices and request the tour guide of their choice, for example, "I want to learn about Impressionism," and send the request to the server.

[0609] Input: User's tour guide request

[0610] Output: Request sent to the server

[0611] Specifically, the user requests a "contemporary art tour guide" and the server receives the request, "I would like to learn about Impressionism."

[0612] Step 6: Request analysis and guided search

[0613] The server analyzes the user's request, searches for the moderator's information in the database, and lists detailed guides to "Impressionism" and provides them to the user.

[0614] Input: User request

[0615] Output: Guide list as search results

[0616] Specifically, the server searches a database for detailed guides on "Impressionism," lists them, and provides them to the user.

[0617] Step 7: View and select guide list

[0618] The user browses the list of guides sent from the server, selects the guide he or she wants, and sends the selected information to the server.

[0619] Input: Guide list from the server

[0620] Output: Selected guide information

[0621] Specifically, the user selects the desired guide from the displayed guides and sends that information to the server.

[0622] Step 8: Review and approve your guide

[0623] The server requests the selected guide to confirm and approve the viewing date and time. If the guide approves, the information is sent to the user to confirm the viewing details.

[0624] Input: User-selected guide information

[0625] Output: Guide approval status, notification to user

[0626] Specifically, the server requests the selected guide to confirm the viewing date and time, and notifies the user after approval.

[0627] Step 9: Notification of viewing start and data loading

[0628] When the viewing start time arrives, the server sends a notification to the user and guide's devices, and also preloads the digital data necessary for the viewing.

[0629] Input: Viewing start time

[0630] Output: Viewing start notification, loaded digital data

[0631] Specifically, the server sends a notification that viewing has begun and loads exhibit information.

[0632] Step 10: Guide operation and explanation

[0633] The guide terminal uses a dedicated application to operate the exhibits while providing commentary, and transmits the commentary and screen operation information to the user in real time.

[0634] Input: Exhibit operated by the guide

[0635] Output: Real-time commentary and control information

[0636] Specifically, the guide operates "Masterpiece A" while providing commentary, and transmits this information to the user in real time.

[0637] Step 11: User Actions and Questions

[0638] The user device allows them to view the virtual space in detail and send questions using the chat function.

[0639] Input: User question

[0640] Output: The question sent to the server

[0641] Specifically, when a user submits a question such as "What is this technique?", the question is sent to the server.

[0642] Step 12: Receiving and answering questions

[0643] The server converts user questions into text, translates them if necessary, and sends them to the guide, who then answers the questions in real time.

[0644] Input: User question

[0645] Output: The textual questions sent to the guide

[0646] Specifically, the server receives the user's question, translates it, and sends it to the guide, who then answers it.

[0647] Step 13: Display the survey and collect responses

[0648] After viewing, the server displays a questionnaire on the user's device and collects responses.

[0649] Input: Survey display request after viewing ends

[0650] Output: Collected survey responses

[0651] Specifically, the server displays a questionnaire on the user's device asking, "What did you think of the tour?" and collects the user's responses.

[0652] Step 14: Analyze and provide feedback

[0653] The server analyzes the collected survey results and comments made during the tour, generates a feedback report, and provides it to the museum.

[0654] Input: Collected survey responses and comments made during the tour

[0655] Output: Feedback report

[0656] Specifically, the server generates a feedback report based on the analysis results and provides it to the museum.

[0657] Step 15: Recommend a product and provide a link to purchase

[0658] The server recommends related museum store products based on the exhibits and artists the user has shown interest in during the tour and provides a purchase link to the user's device.

[0659] Input: Exhibits and artists that the user is interested in

[0660] Output: Recommended products and purchase links

[0661] Specifically, if a user is interested in a replica of "Famous Painting A," related products will be displayed and a purchase link will be provided.

[0662] (Application example 1)

[0663] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0664] In traditional art museums and stores, due to physical distance and time constraints, visitors had to travel to the actual site to view and purchase artworks in detail. Furthermore, due to a lack of guidance tailored to the individual interests of visitors, it was difficult to provide a satisfying experience.

[0665] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0666] In this invention, the server includes means for archiving digital data of artworks, means for displaying the digitized artworks on a terminal device, means for recommending the most suitable guide based on the user's interests, means for conducting an online tour in which the guide explains the artworks, means for supporting real-time two-way communication between the guide and the user, means for archiving digital data of store products, means for recommending the most suitable product guide based on the user's interests, and means for conducting an online tour in which the guide explains and suggests details about the products. This allows visitors to view and purchase artworks and products in detail without being restricted by physical distance or time. Furthermore, by providing guidance tailored to the user's individual interests, a highly satisfying experience can be achieved.

[0667] "Artworks" refers to artworks such as paintings, sculptures, and crafts that are exhibited in museums and galleries.

[0668] "Digital data" refers to information such as images, videos, and text recorded in electronic form.

[0669] "Archiving" refers to the process of storing information and data for a long period of time, organizing and storing it so that it can be easily searched and used later.

[0670] "End-device" refers to a terminal device that is directly operated by the user, and includes smartphones, tablets, PCs, smart glasses, etc.

[0671] A "guide" is someone with the expertise to provide users with detailed information about artworks and products and conduct online tours.

[0672] "Recommendation" means selecting the most suitable guide or product based on the user's interests and proposing it to the user.

[0673] An "online tour" is a virtual tour that takes place in real time over the Internet.

[0674] "Two-way communication" refers to a form of communication in which both parties (guide and user) can exchange information with each other.

[0675] "Products" refers to all items sold in stores, including fashion items, home appliances, stationery, etc.

[0676] "Details" are detailed descriptions or information about a particular subject.

[0677] "Proposal" means introducing solutions or products that meet the user's needs.

[0678] "Store" refers to a physical location or online platform for offering goods or services.

[0679] This invention is a system that digitally archives artworks and store merchandise and provides online guides and tours to users. Specific embodiments of each function will be described below.

[0680] 1. Archiving digital data

[0681] The server receives and temporarily stores product data (images, videos, text, etc.) provided by art objects or stores. When museums or stores upload high-resolution images of specific art objects or products, the server receives this data and stores it in a temporary storage directory. The server then analyzes the digital data and extracts metadata (e.g., artist name, year of creation, technique). This data is converted into an appropriate format and stored in a dedicated database. Indexing allows for quicker searching and filtering later.

[0682] 2. Recommendations based on user interests

[0683] Users access the system from their own devices (smartphones, tablets, smart glasses, etc.) and request guidance on a specific category, product, or artwork. For example, if a user enters a request such as "I'd like to see the latest winter fashion items," this information is sent to the server. The server analyzes this request and searches for guide information in its database. It then creates a list of the most suitable guides and sends this list to the user. The user then selects the desired guide from the displayed list and sends the information to the server.

[0684] 3. Online tours

[0685] When the tour begins, the server sends a notification to the user and guide's devices that the tour is about to begin, and pre-loads the digital data necessary for the tour. The guide uses a dedicated application to operate the products and artworks while providing commentary. At this time, the guide's device sends the commentary and screen operation information to the user in real time via the server. Users can operate the virtual space on their own devices to view the artworks and products in detail. Users can also send questions using the chat function. The server receives questions from users, converts them to text, and sends them to the guide. These questions are translated as needed, making multilingual support possible. The guide answers the questions received from users in real time.

[0686] 4. Secondary Features of the Service

[0687] After the tour ends, the server displays a questionnaire on the user's device and collects their responses. The collected questionnaire results and comments made during the tour are analyzed, and a feedback report is generated and provided to the museum or store. This allows for service improvements. Furthermore, based on the art pieces and products that the user showed interest in during the tour, related products are recommended and a purchasing link is provided to the user's device. For example, if a user is interested in the latest winter fashion items, related products are displayed and a link to purchase them is provided. This provides museums and stores with a new source of revenue and allows users to enjoy a high-quality guided experience from a remote location.

[0688] Adding specific examples

[0689] For example, if a user requests to "see the latest winter fashion items," the server will recommend the most suitable guide, who will then introduce "latest winter fashion item A" and explain its features and materials. Also, if a user asks a question about a product during the tour, the server will convert the question into text and transmit it to the guide in real time, providing an environment where an answer can be obtained immediately.

[0690] Prompt Sentence Examples

[0691] User request: "I want to see the latest winter fashion items"

[0692] Guide's answer: "Here's the latest winter fashion item, item A. It's made by a famous designer and is made from high-quality cashmere and wool. Its unique style and comfort make it a popular item in 2023."

[0693] This invention allows users to enjoy the experience of appreciating and purchasing high-quality artworks and merchandise without being physically restricted, and also provides customized guidance based on the user's individual interests, resulting in a highly satisfying service.

[0694] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0695] Step 1:

[0696] The server receives product data (images, videos, text, etc.) provided by the art works or stores and stores them in a temporary directory. The input is high-resolution images and detailed information provided by the museum or store, and the output is the temporarily stored data.

[0697] Step 2:

[0698] The server analyzes the digital data stored in the temporary storage directory and extracts metadata (artist name, production year, technique, etc.). The input is the stored digital data, and the output is the extracted metadata. Specific operations include using image recognition software and text analysis algorithms.

[0699] Step 3:

[0700] The server converts the analyzed digital data into an appropriate format and stores it in a database. The input is metadata and digital data, and the output is digital data stored in the database. Specifically, it performs data format conversion and indexing.

[0701] Step 4:

[0702] A user accesses the system from a terminal and requests information about a specific category, product, or artwork. The input is the user's request, and the output is the request information being sent to the server. Specific operations include receiving input through the user interface.

[0703] Step 5:

[0704] The server analyzes the user's request and searches for guide information in the database. The input is the user's request information, and the output is a list of the most suitable guides. The specific operation uses a search algorithm based on the request content.

[0705] Step 6:

[0706] The server sends a list of the most suitable guides to the user. The input is the list of guides, and the output is the guide information displayed on the user's terminal. Specific operations include data format conversion and transmission.

[0707] Step 7:

[0708] The user selects the desired guide from the list and sends the information to the server. The input is the selected guide information, and the output is sending the selected information to the server. The specific operation involves operating the selection interface.

[0709] Step 8:

[0710] When the tour start time arrives, the server sends a notification of the tour start to the user and guide's terminals and loads the necessary digital data in advance. The input is the tour start time and digital data, and the output is the notification and the loaded data on the terminal. Specific operations include schedule management and data transmission.

[0711] Step 9:

[0712] The guide uses a dedicated application to operate the products and artworks while providing commentary. The input is the loaded digital data and the guide's commentary, and the output is real-time commentary and screen operation information. Specific operations include voice input and screen operation.

[0713] Step 10:

[0714] Users operate the virtual space on their devices to view artworks and products in detail. They also use the chat function to send questions. The input is the user's operations and questions, and the output is the displayed virtual space and the questions sent. Specific operations include operating the user interface and sending questions.

[0715] Step 11:

[0716] The server receives the user's question, converts it into text, and sends it to the guide. The input is the user's question, and the output is the converted text and the transmitted question. Specific operations include voice recognition software and data transmission.

[0717] Step 12:

[0718] After the tour ends, the server displays a questionnaire on the user's device and collects responses. The input is the questionnaire form and the user's responses, and the output is the collected questionnaire data. Specifically, the server displays the questionnaire form and collects responses.

[0719] Step 13:

[0720] The server analyzes the collected survey results and comments made during the tour, generates a feedback report, and provides it to museums and stores. The input is the survey results and comments, and the output is the generated feedback report. Specific operations include data analysis and report generation.

[0721] Step 14:

[0722] The server recommends related products and provides purchase links based on the artworks and products that the user shows interest in during the tour. The input is the user's interest information and related product data, and the output is the recommended products and purchase links. Specific operations include data mining and link generation.

[0723] In this way, users can enjoy the experience of viewing and purchasing high-quality artworks and merchandise without being restricted by physical constraints, and by providing customized guidance based on individual interests, a highly satisfying service is realized.

[0724] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0725] The present invention is a system that digitally archives museum exhibits and provides a virtual museum that users can access online. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, a more personalized experience is provided. Specific embodiments of each function are described below.

[0726] 1. Digital archiving of museums

[0727] server

[0728] The server receives digital data (images, videos, text, etc.) of exhibits provided by the museum and temporarily stores it. For example, when the museum uploads a high-resolution image of "Famous Painting A," the server receives this data and moves it to a temporary storage directory. The server then analyzes the digital data and extracts metadata (e.g., artist name, year of production, technique used). The server then converts the data into an appropriate format and stores it in a dedicated database. The server also indexes the data to enable quick searches and filtering later.

[0729] 2. Online tour guide matching

[0730] user

[0731] A user accesses the virtual museum from their device and requests a tour guide. For example, if they input their desire to "learn about Impressionism," this information is sent to the server. The server analyzes this request and searches for guide information in its database. For example, it creates a list of guides who are knowledgeable about "Impressionism" and sends this list to the user. The user selects the desired guide from the displayed guides and sends that information to the server. The server then requests confirmation and approval of the tour date and time from the selected guide, and if approval is granted, sends the confirmed information to the user.

[0732] 3. Conducting Virtual Tours

[0733] Server, user terminal, guide terminal

[0734] When the tour begins, the server sends a notification to the user and guide's devices that the tour is about to begin, and pre-loads the digital data necessary for the tour. The guide uses a dedicated application to operate the exhibits while giving their explanation. At this time, the guide's device sends the explanation and screen operation information to the user in real time via the server. Users can operate the virtual space on their own devices and view the exhibits in detail. Users can also send questions using the chat function. The server receives questions from users, converts them to text, and sends them to the guide. These questions are translated as needed, making multilingual support possible. The guide answers the users' questions in real time.

[0735] 4. Secondary Features of the Service

[0736] Server, user terminal

[0737] After the tour ends, the server displays a questionnaire on the user's device and collects feedback information. The collected questionnaire results and comments made during the tour are analyzed, and a feedback report is generated and provided to the museum. Based on the exhibits and artists that the user showed interest in during the tour, the server also generates information recommending products from the museum store and sends this to the user's device. Related products are displayed on the user's device, and a link to purchase them is provided. For example, related products for "Famous Painting A" are displayed, and the user can purchase them via a link to an e-commerce site.

[0738] 5. Incorporating an Emotional Engine

[0739] Emotion Engine

[0740] The emotion engine analyzes the user's facial expression data, tone of voice, and text input data collected in real time from the user's device to recognize the user's emotional state, for example, determining whether the user is excited or bored by a particular exhibit.

[0741] Use of Emotional Data

[0742] Server, guide terminal

[0743] The emotion engine analyzes the user's emotional state and sends it back to the guide terminal in real time via the server. Based on this information, the guide can dynamically adjust the content of the commentary or provide additional information to pique the user's interest. For example, if the user is not interested in a particular exhibit, the guide can introduce other related exhibits.

[0744] Feedback Improvements

[0745] server

[0746] The server generates feedback to improve the user experience based on the emotional data analyzed by the emotion engine and provides it to the museum, which can then use it to improve the exhibits and tour content.

[0747] In this way, the system of the present invention recognizes and reflects user emotions in real time, providing a more personalized and high-quality online museum experience.

[0748] The processing flow will be explained below.

[0749] 1. Digital archiving of museums

[0750] Step 1:

[0751] Server: Provides a dedicated form on the website for museum staff to upload digital data (images, videos, text, etc.) of exhibits.

[0752] Step 2:

[0753] Server: Receives uploaded digital data and moves it to a temporary storage directory. For example, if a museum uploads a high-resolution image of "Famous Painting A," the data is stored in that directory.

[0754] Step 3:

[0755] Server: Analyzes the received digital data and executes analysis scripts to extract metadata (artist name, year of production, technique, etc.). The extracted metadata is stored in a database along with the data.

[0756] Step 4:

[0757] Server: Indexes the data to enable quick searches and filtering later. The indexed data is efficiently managed in a database.

[0758] 2. Online tour guide matching

[0759] Step 1:

[0760] User: The user accesses the virtual museum website from their device, enters the tour guide request screen, and enters "I would like a tour of the Impressionists." This information is sent to the server.

[0761] Step 2:

[0762] Server: Analyzes the user's request, searches for guide information in the database, generates a list of target guides, and sends it to the user's device.

[0763] Step 3:

[0764] User: The user selects the desired guide from the displayed guides and sends the selected information to the server. For example, if the user selects "Guide B," the information is sent to the server.

[0765] Step 4:

[0766] Server: Requests confirmation and approval of the tour date and time from the selected guide, and sends the confirmation information to the user once approval is received.

[0767] 3. Conducting Virtual Tours

[0768] Step 1:

[0769] Server: When the tour start time arrives, the server sends a tour start notification to the user and guide's devices and preloads the digital data required for the tour.

[0770] Step 2:

[0771] Guide: The guide uses a dedicated application to operate the exhibits and provide commentary. Screen operations and commentary are distributed to users in real time via a server.

[0772] Step 3:

[0773] Users: Users can operate the virtual space on their own devices, view the exhibits in detail, and send questions to the guide using the chat function.

[0774] Step 4:

[0775] Server: Converts user questions into text and sends it to the guide. If necessary, translates the questions to support multiple languages.

[0776] Step 5:

[0777] Guide: The guide answers users' questions in real time.

[0778] 4. Secondary Features of the Service

[0779] Step 1:

[0780] Terminal: After the tour ends, the server displays a questionnaire on the user's terminal to collect feedback information. The user answers the questionnaire and sends the results to the server.

[0781] Step 2:

[0782] Server: Analyzes the collected survey results and comments made during the tour, generates a feedback report, and provides it to the museum.

[0783] Step 3:

[0784] Server: Generates information recommending products from the museum store based on the exhibits and artists that the user is interested in during the tour, and sends this information to the user's device.

[0785] Step 4:

[0786] Device: Display related products on the user's device and provide a purchase link. For example, related products for "Famous Painting A" are displayed, and the user can purchase them via a link to an e-commerce site.

[0787] 5. Incorporating an Emotional Engine

[0788] Step 1:

[0789] Device: Obtains facial expression data, tone of voice, text input, etc. in real time from the user's device.

[0790] Step 2:

[0791] Server: Sends the acquired data to the emotion engine to analyze the user's emotional state, for example, analyzing the emotions (excitement, indifference, etc.) when the user is viewing a particular exhibit.

[0792] Step 3:

[0793] Server: Based on the analysis results, the user's emotional state is fed back to the guide device in real time.

[0794] Step 4:

[0795] Guide: The guide dynamically adjusts its commentary based on the emotional feedback it receives, for example providing more detailed information about an exhibit that excites the user.

[0796] Step 5:

[0797] Server: After the tour ends, the emotion engine generates feedback based on the emotional data analyzed and provides it to the museum.

[0798] In this way, the specific processing content at each step ensures that users can have a high-quality experience through the virtual museum. In addition, by incorporating an emotion engine, a more personalized and engaging experience is provided.

[0799] Example 2

[0800] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0801] Conventional digital archive systems for museum exhibits simply store and display digital data of exhibits, limiting the user experience. They also lack digital tour guide recommendations and two-way communication functions, making it difficult to provide a personalized experience tailored to participants. Furthermore, they lack the ability to recognize users' emotions in real time and reflect them in the guide's commentary, making it difficult to respond flexibly to users' interests and level of understanding.

[0802] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for digitally archiving museum exhibits, means for displaying the digitized exhibits on a user terminal, means for recommending an optimal tour guide based on the user's interests and concerns, means for conducting an online tour in which the guide explains the exhibits, means for supporting two-way communication between the guide and the user in real time, means for analyzing the user's emotional data and providing feedback to the guide in real time, and means for collecting feedback from the user after the tour ends and providing the feedback to the museum. This allows the user to experience a high-quality virtual tour of the museum's exhibits through the digital archive, and receive a personalized explanation based on the real-time emotional data.

[0803] An "art museum" is a facility that displays exhibits for the general public and aims to promote education and culture.

[0804] "Exhibits" refers to cultural assets and artworks such as paintings, sculptures, crafts, and photographs displayed in art museums.

[0805] "Digital archiving" is the process of recording, preserving, and managing exhibits as digital data (images, videos, text, etc.).

[0806] "User Terminal" refers to a device, such as a computer, smartphone, or tablet, that a User uses to access the System via the Internet.

[0807] A "tour guide" is a professional or function that provides commentary and explanations about museum exhibits, either online or in person.

[0808] The "recommendation method" is part of a system that uses algorithms and filtering functions to select and suggest the most suitable tour guide based on the user's interests and concerns.

[0809] An "online tour" is a service that allows users to virtually view museum exhibits using the Internet while receiving real-time commentary from a guide.

[0810] "Two-way communication" refers to the interactive function that allows users and guides to exchange information in real time.

[0811] "Emotional data" refers to information about a user's psychological state and emotions analyzed from facial expressions, tone of voice, text input, etc.

[0812] "Feedback" refers to information such as opinions, impressions, and evaluations collected from users after the tour, and is used to improve and evaluate the service.

[0813] The present invention is a system that digitally archives museum exhibits and provides a virtual museum that users can access online. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system provides a more personalized experience. Specific embodiments of each function are described below.

[0814] Digital archiving of museum exhibits

[0815] server

[0816] The server receives digital data (images, videos, text, etc.) of exhibits provided by the museum and temporarily stores it. For example, if the museum uploads a high-resolution image of "Famous Painting A," the server receives this data and moves it to a temporary storage directory. The server analyzes the received digital data and extracts metadata (artist name, year of production, technique, etc.). Software such as image analysis algorithms is used for the analysis. The analyzed data is converted into an appropriate format and stored in a dedicated database. In addition, indexing the data allows for faster searching and filtering later.

[0817] Online tour guide matching

[0818] user

[0819] A user accesses the virtual museum from their device and requests a tour guide. For example, if they enter their request as "I would like to learn about Impressionism," this information is sent to the server. The server analyzes the user's request and searches for guide information in its database. For example, it creates a list of guides who are knowledgeable about "Impressionism" and sends this list to the user. The user selects the desired guide from the displayed guides and sends that information to the server. The server then requests confirmation and approval of the tour date and time from the selected guide, and if approval is granted, sends the confirmed information to the user.

[0820] Conducting a virtual tour

[0821] Server, user terminal, guide terminal

[0822] When the tour begins, the server sends a notification to the user's and guide's devices and pre-loads the digital data necessary for the tour. The guide uses a dedicated application to operate the exhibits while providing commentary. For example, the guide might display details of "Masterpiece A" on the screen while explaining its background and techniques. The guide's device sends the commentary and screen operation information to the user in real time via the server. Users can operate the virtual space on their own devices and view the exhibits in detail. Users can also send questions using the chat function. The server receives questions from users, converts them to text, and sends them to the guide. These questions are translated as needed, making multilingual support possible. The guide then answers the user's questions in real time.

[0823] Service Sub-Features

[0824] Server, user terminal

[0825] After the tour ends, the server displays a questionnaire on the user's device to collect feedback information. For example, feedback is collected by having the user answer the question, "How satisfied were you with this tour?" The collected feedback information is analyzed and provided to the museum. The server analyzes the collected feedback and the user's interests, generates information recommending related museum store products, and sends this information to the user's device. For example, related products for "Famous Painting A" are displayed, and the user can purchase them via a link to an e-commerce site.

[0826] Incorporating an emotion engine

[0827] Emotion engine, server

[0828] The emotion engine analyzes the user's facial expression data, tone of voice, and text input content obtained in real time from the user's device to recognize the user's emotional state. For example, it determines whether the user is excited or bored by a particular exhibit. This information is fed back to the guide device via the server. The guide then dynamically adjusts the content of the commentary based on this information. For example, if the user shows no interest in a particular exhibit, the guide will introduce other related exhibits. The server generates feedback to improve the user experience based on the emotional data analyzed by the emotion engine and provides it to the museum.

[0829] Examples of concrete examples and prompts

[0830] For example, if an art museum were to digitally archive an "Impressionist Exhibition," it would follow the steps below.

[0831] 1. The museum uploads a high-resolution image of "Masterpiece A" to the server.

[0832] 2. The server analyzes the image data and extracts metadata such as the artist's name and technique.

[0833] 3. The user requests, "I want to learn about Impressionism," and the server matches them with a suitable tour guide.

[0834] 4. When the tour start time arrives, the server notifies the user and guide devices and the virtual tour begins.

[0835] 5. The guide explains the details of the exhibits, and when the user asks questions, they are sent to the guide via the server.

[0836] 6. After the tour, a survey will be displayed on the user's device to collect feedback information.

[0837] Prompt Sentence Examples

[0838] I want to book a virtual tour about Impressionism.

[0839] I want to know more about Masterpiece A.

[0840] The above is an embodiment of the present invention, which allows users to experience a high-quality virtual tour and receive personalized commentary in real time.

[0841] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0842] Step 1:

[0843] The server receives the digital data from the museum.

[0844] Input: High-resolution images, videos, and text data uploaded by museums.

[0845] Data processing: The input data is saved in a temporary directory.

[0846] Output: Digital data saved in temporary directory.

[0847] Specific operation: When the museum uploads a high-resolution image of "Famous Painting A," the server receives this data and moves it to a temporary storage directory.

[0848] Step 2:

[0849] The server analyzes the received digital data and extracts the metadata.

[0850] Input: Digital data stored in the temporary directory.

[0851] Data processing: Analyze the digital data using software such as image analysis algorithms.

[0852] Output: Extracted metadata (artist name, year of production, technique, etc.).

[0853] Specific operation: The server analyzes the stored image data and automatically extracts information about the artist and era of the work.

[0854] Step 3:

[0855] The server converts the parsed data into an appropriate format and stores it in a database.

[0856] Input: Extracted metadata and original digital data.

[0857] Data processing: Image data is converted into a specific compressed format and stored in a database along with metadata.

[0858] Output: Digital data and metadata stored in a database.

[0859] Specific operation: The server converts the analyzed data into an appropriate format and stores the digital data and metadata in a database.

[0860] Step 4:

[0861] Users access the virtual museum from their own devices and request a tour guide.

[0862] Input: A tour guide request sent from the user's device (e.g., "I want to learn about Impressionism").

[0863] Data processing: The server receives and analyzes the user's request.

[0864] Output: The tour guide request sent to the server.

[0865] What it does: A user types in "I want to learn about Impressionism," and this information is sent to the server.

[0866] Step 5:

[0867] The server analyzes the tour guide request and matches the most suitable guide.

[0868] Input: User request data.

[0869] Data processing: The server searches the guide information in the database and lists related guides.

[0870] Output: A list of guides that will be sent to the user.

[0871] What it does: The server compiles a list of detailed guides to "Impressionism" from its database and sends this list to the user.

[0872] Step 6:

[0873] The user selects the desired guide and submits it to the server, which requests confirmation and approval of the tour date and time.

[0874] Input: User selected guide information.

[0875] Data processing: The server confirms the tour date and time with the selected guide and requests approval.

[0876] Output: Tour information approved by the guide.

[0877] Specific operation: The user selects the desired guide and sends the information to the server. The server requests the guide to confirm and approve the tour date and time, and once approval is received, the confirmation information is sent to the user.

[0878] Step 7:

[0879] At the start time of the tour, the server sends a notification and pre-loads the digital data required for the tour.

[0880] Input: Confirmed tour information.

[0881] Data processing: Preload the necessary digital data.

[0882] Output: Tour start notification to user and guide devices.

[0883] Specific operation: When the tour start time arrives, the server sends a notification and loads the digital data required for the tour into the user's and guide's devices in advance.

[0884] Step 8:

[0885] The guide will provide commentary while operating the exhibits, and users can send questions via their terminal.

[0886] Input: Data about the exhibits operated by the guide, user question data.

[0887] Data processing: The server sends the commentary and screen operation information to the user, and sends the user's questions to the guide.

[0888] Output: Instructions sent to the user, user questions sent to the guide.

[0889] Specific operation: The guide explains "Famous Painting A," and when the user asks a question, the server sends the question to the guide, who then answers in real time.

[0890] Step 9:

[0891] The emotion engine analyzes the user's emotional data and provides feedback to the guide via the server.

[0892] Input: User facial expression data, tone of voice, and text input.

[0893] Data processing: The emotion engine analyzes the emotion data and provides feedback to the guide via the server.

[0894] Output: The user's emotional state, which is fed back to the guide.

[0895] Specific operation: The emotion engine analyzes the user's emotions and provides the guide with the user's excitement level or boredom level in real time.

[0896] Step 10:

[0897] After the tour ends, the server displays a questionnaire on the user's device to collect feedback information.

[0898] Input: User survey data after the tour.

[0899] Data processing: Analyze the collected survey results and generate a feedback report to provide to the museum.

[0900] Output: Feedback report provided to the museum, related product recommendations displayed on the user's device.

[0901] Specific operation: After the tour ends, a questionnaire is displayed on the user's device, and feedback information is collected and analyzed. The analysis results are provided to the museum, and related product recommendations are displayed on the user's device.

[0902] (Application example 2)

[0903] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0904] Conventional online virtual tour systems for museums and stores have struggled to provide a personalized experience based on the user's interests and emotions. Furthermore, they lacked the functionality to analyze the user's facial expressions and tone of voice to provide real-time product recommendations and information. This resulted in low user satisfaction.

[0905] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for digitalizing exhibits, means for displaying the digitalized exhibits on a user terminal, means for recommending an optimal guide based on the user's interests and concerns, means for conducting an online guided tour in which the guide explains the exhibits, means for supporting two-way communication between the guide and the user in real time, means for analyzing the user's facial expression data and voice data in real time to recognize the user's emotional state, and means for recommending products that will interest the user and providing additional information based on the user's emotional state. This enables a personalized and high-quality online experience that is tailored to the user's emotions and interests.

[0906] "Means of digitalizing exhibits" refers to the technology used to convert physical exhibits into digital data and store it in a database.

[0907] "Means for displaying digitalized exhibits on user terminals" refers to technology for displaying exhibits stored as digital data on terminals used by users.

[0908] "Means for recommending the most suitable guide based on the user's interests and concerns" refers to technology that analyzes the interests and concerns expressed by the user and automatically selects and recommends the guide that best suits them.

[0909] "Means for conducting online guided tours in which a guide explains exhibits" refers to technology that enables a guide to provide explanations about digitized exhibits and share the content with users in real time.

[0910] "Means for supporting two-way communication between a guide and a user in real time" refers to technology that enables a guide and a user to have two-way communication in real time.

[0911] "Means for analyzing a user's facial expression data and voice data in real time to recognize their emotional state" refers to technology that uses a camera or microphone to capture the user's facial expressions and voice, analyzes that data, and recognizes the user's emotional state in real time.

[0912] The "means for recommending products and providing additional information that may interest a user based on the user's emotional state" is a technology that dynamically provides products and related information that may be of interest to a user based on the user's recognized emotional state.

[0913] This invention is a system that allows users to experience exhibits and products in a virtual space, and provides personalized information by recognizing the user's emotions in real time. This system is constructed mainly using a server, a user terminal, and a guide terminal, and specific embodiments are described below.

[0914] Hardware and Software Use

[0915] The server is installed on the cloud and performs temporary data storage, analysis, and database management. Smartphones, tablets, PCs, etc. are used as user and guide devices, and these devices must be equipped with a camera and microphone. The main software used is as follows:

[0916] Google Cloud Vision API: A service for emotion analysis that analyzes image data and recognizes the user's facial expressions.

[0917] Firebase: A database service that processes and stores data in real time, managing user sentiment data and feedback information.

[0918] WebRTC: A technology that enables real-time communication, used for video chat and data streaming between users and guides.

[0919] Data acquisition and analysis

[0920] The server receives and temporarily stores digital data (images, videos, text, etc.) of exhibits and products provided by museums and stores. When high-resolution image data and detailed information are uploaded, the server analyzes it, extracts appropriate metadata (e.g., product name, price, description), and stores it in a database.

[0921] User information analysis and matching

[0922] When a user visits the virtual store, the user's device captures facial expression data using a camera and voice data using a microphone. This data is sent in real time to the Google Cloud Vision API, which analyzes the user's emotional state. The analysis results are stored in Firebase and fed back to the guide via the server.

[0923] Online tours and guided tours

[0924] When the tour begins, the server sends a notification to the user's and guide's devices and pre-loads the digital data necessary for the tour. The guide uses a dedicated application to interact with the exhibits and products while providing commentary, which is then sent to the user's device in real time. The user can move their viewpoint in any direction within the virtual space to view the exhibits in detail. They can also use the chat function to send questions to the guide.

[0925] Feedback and recommendation system

[0926] After the tour ends, the server displays a questionnaire screen on the user's device to collect feedback. Based on this feedback information and emotional data obtained during the tour, the server generates recommendation information for products and exhibits that the user showed interest in and provides it to the user's device. This information also includes links to related products and services, allowing users to smoothly make purchases or check details.

[0927] Examples of concrete examples and prompts

[0928] As a concrete example, if a user is exploring the wearable devices section of a virtual store and their smartphone camera captures their facial expression and detects "excitement," the system can analyze this data and automatically display more information and reviews related to the product they are particularly interested in (e.g., the latest smartwatch).

[0929] Example prompt sentence:

[0930] "Analyze users' facial expressions and tone of voice in real time and make appropriate product recommendations based on their emotional state (e.g., joy, anger, sadness, happiness, etc.). Use Firebase and Google Cloud Vision API to store user emotion data and notify the shopping guide in real time."

[0931] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0932] Step 1:

[0933] The server receives and temporarily stores digital data (e.g., images, videos, text) of exhibits and products provided by museums and stores. It then analyzes this data to extract metadata (e.g., product name, price, description) and stores it in a database. The input is digital data, and the output is metadata.

[0934] Step 2:

[0935] The user device captures the user's facial expression data using a camera and voice data using a microphone. This data is sent in real time to the Google Cloud Vision API, which analyzes the user's emotional state. The input is the user's facial expression and voice data, and the output is the analysis result of the emotional state.

[0936] Step 3:

[0937] The server stores the emotional state data obtained as a result of the analysis in Firebase, and uses this data to generate recommendation data for products that will interest the user. The input is the emotional state analysis results, and the output is recommendation data.

[0938] Step 4:

[0939] When a user accesses a virtual store, the server recommends the most suitable guide based on the user's interests. Based on input data about interests obtained from the user's device, the server searches for guide information in the database and lists the most suitable guides. The input is data about the user's interests, and the output is a list of guides.

[0940] Step 5:

[0941] At the start of the tour, the server sends notifications to the user and guide's devices and preloads digital data about the virtual store. The user and guide start a real-time video chat using WebRTC, and the server relays the communication between them. The input is a connection request from both devices, and the output is the establishment of the connection.

[0942] Step 6:

[0943] During the tour, the guide explains the exhibits and products, and the content is sent to the user in real time via the server. Users can use the chat function to send questions, which the server converts into text and sends to the guide. The input is the guide's explanation data and the user's question data, and the output is the textual question data and explanation content.

[0944] Step 7:

[0945] After the tour ends, the server displays a questionnaire screen on the user's device to collect feedback. The collected feedback information and emotional data collected during the tour are analyzed, and recommendation information is generated for products and exhibits that the user showed interest in. The input is feedback data and emotional data, and the output is recommendation information.

[0946] Step 8:

[0947] The server sends the final recommendation information to the user's device and provides detailed information and a purchase link for related products and services, allowing the user to smoothly purchase products or check details. The input is the generated recommendation information, and the output is the detailed information and purchase link displayed on the user's device.

[0948] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0949] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0950] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0951] [Third embodiment]

[0952] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0953] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0954] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0955] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0956] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0957] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0958] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0959] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0960] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0961] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0962] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0963] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0964] The present invention provides a system that digitally archives art museum exhibits and provides a virtual museum that users can access online. Specific embodiments of each function are described below.

[0965] 1. Digital archiving of museums

[0966] server

[0967] The server receives digital data (images, videos, text, etc.) of exhibits provided by the museum and temporarily stores it. For example, when the museum uploads a high-resolution image of "Famous Painting A," the server receives this data and stores it in a temporary storage directory. The server then analyzes the digital data and extracts metadata (e.g., artist name, year of production, technique used). The server then converts the data into an appropriate format and stores it in a dedicated database. The server also indexes the data to enable quick searches and filtering later.

[0968] 2. Online tour guide matching

[0969] user

[0970] A user accesses the virtual museum from their device and requests a tour guide. For example, if they input their desire to "learn about Impressionism," this information is sent to the server. The server analyzes this request and searches for guide information in its database. For example, it creates a list of guides who are knowledgeable about "Impressionism" and sends this list to the user. The user selects the desired guide from the displayed guides and sends that information to the server. The server then requests confirmation and approval of the tour date and time from the selected guide, and if approval is granted, sends the confirmed information to the user.

[0971] 3. Conducting Virtual Tours

[0972] Server, user terminal, guide terminal

[0973] When the tour begins, the server sends a notification to the user and guide's devices that the tour is about to begin, and pre-loads the digital data necessary for the tour. The guide uses a dedicated application to operate the exhibits while giving their explanation. At this time, the guide's device sends the explanation and screen operation information to the user in real time via the server. Users can operate the virtual space on their own devices and view the exhibits in detail. Users can also send questions using the chat function. The server receives questions from users, converts them to text, and sends them to the guide. These questions are translated as needed, making multilingual support possible. The guide answers the users' questions in real time.

[0974] 4. Secondary Features of the Service

[0975] Server, user terminal

[0976] After the tour ends, the server displays a survey on the user's device and collects responses. The collected survey results and comments made during the tour are analyzed, and a feedback report is generated and provided to the museum. This allows the museum to improve its services. Furthermore, based on the exhibits and artists that the user showed interest in during the tour, the server recommends products from the museum store and provides a purchasing link to the user's device. For example, if a user is interested in a replica of "Masterpiece A," related products will be displayed and a link to purchase them will be provided.

[0977] This will provide museums with new revenue streams and allow visitors to have a high-quality guided museum experience remotely.

[0978] The processing flow will be explained below.

[0979] 1. Digital archiving of museums

[0980] Step 1:

[0981] Server: Provides a dedicated upload form to receive digital data (images, videos, text, etc.) of exhibits provided by the museum. When a museum staff member uploads a high-resolution image of "Masterpiece A," the server receives this data and moves it to a directory for temporary storage.

[0982] Step 2:

[0983] Server: Analyzes the received digital data and runs analysis scripts to automatically extract metadata (artist name, year of production, technique, etc.). Once the metadata is extracted, the server stores this data in a digital archive database.

[0984] Step 3:

[0985] Server: Indexes the data to enable quick searches and filtering later. The indexed data is efficiently managed in a database.

[0986] 2. Online tour guide matching

[0987] Step 1:

[0988] Device: The user accesses the virtual museum website from their device, enters the tour guide request screen, and enters "I would like a tour of the Impressionists." This information is sent to the server.

[0989] Step 2:

[0990] Server: Analyzes the user's request, searches for guide information in the database, generates a list of target guides, and sends it to the user's device.

[0991] Step 3:

[0992] Terminal: The user selects the desired guide from the displayed guides and sends the selected information to the server. For example, if "Guide B" is selected, the information is sent to the server.

[0993] Step 4:

[0994] Server: Requests confirmation and approval of the tour date and time from the selected guide, and sends the confirmation information to the user once approval is received.

[0995] 3. Conducting Virtual Tours

[0996] Step 1:

[0997] Server: When the tour start time arrives, the server sends a tour start notification to the user and guide's devices and preloads the digital data required for the tour.

[0998] Step 2:

[0999] Guide: The guide uses a dedicated application to operate the exhibits and provide commentary. Screen operations and commentary are distributed to users in real time via a server.

[1000] Step 3:

[1001] Users: Users can operate the virtual space on their own devices, view the exhibits in detail, and send questions to the guide using the chat function.

[1002] Step 4:

[1003] Server: Converts user questions into text and sends it to the guide. If necessary, translates the questions to support multiple languages.

[1004] Step 5:

[1005] Guide: The guide answers users' questions in real time.

[1006] 4. Secondary Features of the Service

[1007] Step 1:

[1008] Terminal: After the tour ends, the server displays a questionnaire on the user's terminal to collect feedback information. The user answers the questionnaire and sends the results to the server.

[1009] Step 2:

[1010] Server: Analyzes the collected survey results and comments made during the tour, generates a feedback report, and provides it to the museum.

[1011] Step 3:

[1012] Server: Generates information recommending products from the museum store based on the exhibits and artists that the user is interested in during the tour, and sends this information to the user's device.

[1013] Step 4:

[1014] Device: Display related products on the user's device and provide a purchase link. For example, related products for "Famous Painting A" are displayed, and the user can purchase them via a link to an e-commerce site.

[1015] In this way, through the specific processing content at each step, users can have a high-quality experience through the virtual museum.

[1016] Example 1

[1017] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1018] There is a need for a system that allows general visitors to easily and effectively view cultural assets in museums. However, existing systems have issues such as insufficient digital archiving and the inability to communicate in real time between users and guides. Furthermore, they lack the ability to provide a personalized experience based on users' interests, which leads to a decline in user satisfaction.

[1019] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1020] In this invention, the server includes means for digitally archiving the museum's cultural assets, means for displaying the digitized cultural assets on an information processing terminal, and means for recommending the most appropriate tour guide based on the user's interests. This enables digital archiving of cultural assets and personalized online viewing based on the archived data. The server also includes means for extracting metadata from received digital data, converting it into a storage format, and storing it in a database; means for analyzing the user's viewing request and searching for tour guide information in the database; means for requesting the tour guide to confirm and approve the date and time of the viewing; means for sending a tour start notification and preloading the necessary digital data; and means for receiving user operations and questions during the viewing of the cultural assets, converting them into text, translating them into multiple languages, and sending them to the tour guide. This enables real-time two-way communication between the user and the tour guide, providing a richer online viewing experience.

[1021] An "art museum" is a facility that collects, preserves, exhibits, and makes available to the public cultural assets and works of art.

[1022] "Cultural properties" refer to items, documents, and exhibits that have historical, cultural, or academic value.

[1023] "Digital archiving" is the process of preserving physical cultural assets as digital data to increase their convenience.

[1024] "Information processing terminal" refers to a device such as a computer, tablet, or smartphone that displays and manipulates digital data.

[1025] "User" means an individual or group who uses the system to view digitally archived cultural property.

[1026] A "facilitator" is an expert or guide who provides explanations about cultural properties to users on the system.

[1027] "Online viewing" is the experience of viewing digitally archived cultural assets via the Internet.

[1028] "Two-way communication" is a function that allows users and moderators to exchange messages and information in real time.

[1029] "Metadata" is additional information about digital data, such as the artist's name, the year of production, and the technique used.

[1030] A "database" is a system for efficiently storing, managing, and searching multiple digital data.

[1031] "Multilingual translation" is a function that translates into different languages ​​and supports communication between users and facilitators who speak different languages.

[1032] The present invention is a system that digitally archives cultural assets in art museums and provides a virtual museum that users can access online. The system of the present invention operates primarily on a server and functions in conjunction with information processing terminals (user terminals and guide terminals). Specific embodiments of each function are described below.

[1033] server

[1034] The server receives and temporarily stores digital data (images, videos, text, etc.) of cultural assets provided by the museum. At this time, the server analyzes the digital data and uses image processing algorithms and natural language processing technology to extract metadata (artist name, year of production, technique, etc.). The server then converts the digital data into an appropriate storage format and stores it in a dedicated database. The server also indexes the data, making it quickly searchable.

[1035] For example, if a museum provides a high-resolution image of "Famous Painting A," the server stores the image in a temporary directory and uses image processing algorithms to extract the artist's name and the year of production. The server then converts the image along with the extracted metadata into an appropriate format and stores it in the database.

[1036] user

[1037] A user accesses the virtual museum from their device and requests a tour guide. For example, if the user inputs their desire to "learn about Impressionism," this information is sent to the server. The server analyzes this request and searches for guide information in its database. The search results list guides who are knowledgeable about Impressionism and provides this list to the user. The user selects the desired guide and requests confirmation and approval of the tour date and time from the server.

[1038] Specifically, when a user requests a "tour guide of contemporary art," the server searches the database for guides knowledgeable about contemporary art and presents a list to the user. The user selects the desired guide and sends the information to the server.

[1039] Guide terminal

[1040] When the tour begins, the server sends a notification to the user's and guide's devices. The server also pre-loads the digital data necessary for the tour. The guide uses a dedicated application to operate the exhibits while providing commentary on the cultural properties. The content of this commentary and screen operation information are sent to the user in real time via the server.

[1041] For example, while the guide is explaining "Famous Painting A," the user may ask a question via chat, such as, "I'd like to know about the techniques used in this painting." The question is sent to the guide via the server and translated as necessary. The guide can then answer the question in real time.

[1042] Server and user terminal

[1043] After the tour ends, the server displays a questionnaire on the user's device and collects responses. The server analyzes the collected questionnaire results and comments made during the tour to generate a feedback report and provide it to the museum, providing specific data for service improvement. Furthermore, based on the cultural assets the user showed interest in during the tour, the server recommends related museum store products to the user's device and provides a link to purchase them.

[1044] For example, if a user is interested in a replica of "Famous Painting A," the system will display related products and provide a link to purchase them, enabling museums to secure new revenue streams and providing visitors with a high-quality, guided museum experience from a remote location.

[1045] Example prompts for generative AI models

[1046] "Please detail the steps a user can take to request an Impressionist tour guide in your virtual museum system."

[1047] "Please explain in detail how the server receives and stores data from the museum in the digital archive system."

[1048] "Please detail the steps your server takes to display and collect a survey from users after the tour has finished."

[1049] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1050] Step 1: Receiving and temporarily storing digital data

[1051] The server receives digital data of cultural assets (images, videos, text, etc.) from the museum and stores the received data in a temporary directory.

[1052] Input: Digital data provided by the museum

[1053] Output: Digital data stored in a temporary directory

[1054] Specifically, when the museum uploads a high-resolution image file of "Masterpiece A," the server temporarily stores this data in a storage directory.

[1055] Step 2: Extracting Metadata

[1056] The server analyzes the digital data stored in the temporary storage directory and extracts metadata, such as the artist's name, production year, and technique, using image processing algorithms and natural language processing technology.

[1057] Input: Digital data stored in a temporary directory

[1058] Output: Extracted metadata

[1059] Specifically, the server automatically recognizes the artist's name, year of production, and techniques used from the image of "Masterpiece A" and extracts them as metadata.

[1060] Step 3: Convert and save the data format

[1061] The server converts the digital data along with the extracted metadata into a designated storage format, and stores the converted data in a dedicated database.

[1062] Input: Digital data, extracted metadata

[1063] Output: Data converted to a suitable storage format, data stored in a database

[1064] Specifically, it converts large image files into a format suitable for web display and stores the converted data and metadata in a database.

[1065] Step 4: Indexing

[1066] The server indexes the stored data, which allows for fast searches.

[1067] Input: Data stored in a database

[1068] Output: Indexed database

[1069] Specifically, the server adds metadata such as the artist's name and technique to the index, making it easy to search later.

[1070] Step 5: Request a tour guide

[1071] Users access the virtual museum from their own devices and request the tour guide of their choice, for example, "I want to learn about Impressionism," and send the request to the server.

[1072] Input: User's tour guide request

[1073] Output: Request sent to the server

[1074] Specifically, the user requests a "contemporary art tour guide" and the server receives the request, "I would like to learn about Impressionism."

[1075] Step 6: Request analysis and guided search

[1076] The server analyzes the user's request, searches for the moderator's information in the database, and lists detailed guides to "Impressionism" and provides them to the user.

[1077] Input: User request

[1078] Output: Guide list as search results

[1079] Specifically, the server searches a database for detailed guides on "Impressionism," lists them, and provides them to the user.

[1080] Step 7: View and select guide list

[1081] The user browses the list of guides sent from the server, selects the guide he or she wants, and sends the selected information to the server.

[1082] Input: Guide list from the server

[1083] Output: Selected guide information

[1084] Specifically, the user selects the desired guide from the displayed guides and sends that information to the server.

[1085] Step 8: Review and approve your guide

[1086] The server requests the selected guide to confirm and approve the viewing date and time. If the guide approves, the information is sent to the user to confirm the viewing details.

[1087] Input: User-selected guide information

[1088] Output: Guide approval status, notification to user

[1089] Specifically, the server requests the selected guide to confirm the viewing date and time, and notifies the user after approval.

[1090] Step 9: Notification of viewing start and data loading

[1091] When the viewing start time arrives, the server sends a notification to the user and guide's devices, and also preloads the digital data necessary for the viewing.

[1092] Input: Viewing start time

[1093] Output: Viewing start notification, loaded digital data

[1094] Specifically, the server sends a notification that viewing has begun and loads exhibit information.

[1095] Step 10: Guide operation and explanation

[1096] The guide terminal uses a dedicated application to operate the exhibits while providing commentary, and transmits the commentary and screen operation information to the user in real time.

[1097] Input: Exhibit operated by the guide

[1098] Output: Real-time commentary and control information

[1099] Specifically, the guide operates "Masterpiece A" while providing commentary, and transmits this information to the user in real time.

[1100] Step 11: User Actions and Questions

[1101] The user device allows them to view the virtual space in detail and send questions using the chat function.

[1102] Input: User question

[1103] Output: The question sent to the server

[1104] Specifically, when a user submits a question such as "What is this technique?", the question is sent to the server.

[1105] Step 12: Receiving and answering questions

[1106] The server converts user questions into text, translates them if necessary, and sends them to the guide, who then answers the questions in real time.

[1107] Input: User question

[1108] Output: The textual questions sent to the guide

[1109] Specifically, the server receives the user's question, translates it, and sends it to the guide, who then answers it.

[1110] Step 13: Display the survey and collect responses

[1111] After viewing, the server displays a questionnaire on the user's device and collects responses.

[1112] Input: Survey display request after viewing ends

[1113] Output: Collected survey responses

[1114] Specifically, the server displays a questionnaire on the user's device asking, "What did you think of the tour?" and collects the user's responses.

[1115] Step 14: Analyze and provide feedback

[1116] The server analyzes the collected survey results and comments made during the tour, generates a feedback report, and provides it to the museum.

[1117] Input: Collected survey responses and comments made during the tour

[1118] Output: Feedback report

[1119] Specifically, the server generates a feedback report based on the analysis results and provides it to the museum.

[1120] Step 15: Recommend a product and provide a link to purchase

[1121] The server recommends related museum store products based on the exhibits and artists the user has shown interest in during the tour and provides a purchase link to the user's device.

[1122] Input: Exhibits and artists that the user is interested in

[1123] Output: Recommended products and purchase links

[1124] Specifically, if a user is interested in a replica of "Famous Painting A," related products will be displayed and a purchase link will be provided.

[1125] (Application example 1)

[1126] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1127] In traditional art museums and stores, due to physical distance and time constraints, visitors had to travel to the actual site to view and purchase artworks in detail. Furthermore, due to a lack of guidance tailored to the individual interests of visitors, it was difficult to provide a satisfying experience.

[1128] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1129] In this invention, the server includes means for archiving digital data of artworks, means for displaying the digitized artworks on a terminal device, means for recommending the most suitable guide based on the user's interests, means for conducting an online tour in which the guide explains the artworks, means for supporting real-time two-way communication between the guide and the user, means for archiving digital data of store products, means for recommending the most suitable product guide based on the user's interests, and means for conducting an online tour in which the guide explains and suggests details about the products. This allows visitors to view and purchase artworks and products in detail without being restricted by physical distance or time. Furthermore, by providing guidance tailored to the user's individual interests, a highly satisfying experience can be achieved.

[1130] "Artworks" refers to artworks such as paintings, sculptures, and crafts that are exhibited in museums and galleries.

[1131] "Digital data" refers to information such as images, videos, and text recorded in electronic form.

[1132] "Archiving" refers to the process of storing information and data for a long period of time, organizing and storing it so that it can be easily searched and used later.

[1133] "End-device" refers to a terminal device that is directly operated by the user, and includes smartphones, tablets, PCs, smart glasses, etc.

[1134] A "guide" is someone with the expertise to provide users with detailed information about artworks and products and conduct online tours.

[1135] "Recommendation" means selecting the most suitable guide or product based on the user's interests and proposing it to the user.

[1136] An "online tour" is a virtual tour that takes place in real time over the Internet.

[1137] "Two-way communication" refers to a form of communication in which both parties (guide and user) can exchange information with each other.

[1138] "Products" refers to all items sold in stores, including fashion items, home appliances, stationery, etc.

[1139] "Details" are detailed descriptions or information about a particular subject.

[1140] "Proposal" means introducing solutions or products that meet the user's needs.

[1141] "Store" refers to a physical location or online platform for offering goods or services.

[1142] This invention is a system that digitally archives artworks and store merchandise and provides online guides and tours to users. Specific embodiments of each function will be described below.

[1143] 1. Archiving digital data

[1144] The server receives and temporarily stores product data (images, videos, text, etc.) provided by art objects or stores. When museums or stores upload high-resolution images of specific art objects or products, the server receives this data and stores it in a temporary storage directory. The server then analyzes the digital data and extracts metadata (e.g., artist name, year of creation, technique). This data is converted into an appropriate format and stored in a dedicated database. Indexing allows for quicker searching and filtering later.

[1145] 2. Recommendations based on user interests

[1146] Users access the system from their own devices (smartphones, tablets, smart glasses, etc.) and request guidance on a specific category, product, or artwork. For example, if a user enters a request such as "I'd like to see the latest winter fashion items," this information is sent to the server. The server analyzes this request and searches for guide information in its database. It then creates a list of the most suitable guides and sends this list to the user. The user then selects the desired guide from the displayed list and sends the information to the server.

[1147] 3. Online tours

[1148] When the tour begins, the server sends a notification to the user and guide's devices that the tour is about to begin, and pre-loads the digital data necessary for the tour. The guide uses a dedicated application to operate the products and artworks while providing commentary. At this time, the guide's device sends the commentary and screen operation information to the user in real time via the server. Users can operate the virtual space on their own devices to view the artworks and products in detail. Users can also send questions using the chat function. The server receives questions from users, converts them to text, and sends them to the guide. These questions are translated as needed, making multilingual support possible. The guide answers the questions received from users in real time.

[1149] 4. Secondary Features of the Service

[1150] After the tour ends, the server displays a questionnaire on the user's device and collects their responses. The collected questionnaire results and comments made during the tour are analyzed, and a feedback report is generated and provided to the museum or store. This allows for service improvements. Furthermore, based on the art pieces and products that the user showed interest in during the tour, related products are recommended and a purchasing link is provided to the user's device. For example, if a user is interested in the latest winter fashion items, related products are displayed and a link to purchase them is provided. This provides museums and stores with a new source of revenue and allows users to enjoy a high-quality guided experience from a remote location.

[1151] Adding specific examples

[1152] For example, if a user requests to "see the latest winter fashion items," the server will recommend the most suitable guide, who will then introduce "latest winter fashion item A" and explain its features and materials. Also, if a user asks a question about a product during the tour, the server will convert the question into text and transmit it to the guide in real time, providing an environment where an answer can be obtained immediately.

[1153] Prompt Sentence Examples

[1154] User request: "I want to see the latest winter fashion items"

[1155] Guide's answer: "Here's the latest winter fashion item, item A. It's made by a famous designer and is made from high-quality cashmere and wool. Its unique style and comfort make it a popular item in 2023."

[1156] This invention allows users to enjoy the experience of appreciating and purchasing high-quality artworks and merchandise without being physically restricted, and also provides customized guidance based on the user's individual interests, resulting in a highly satisfying service.

[1157] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1158] Step 1:

[1159] The server receives product data (images, videos, text, etc.) provided by the art works or stores and stores them in a temporary directory. The input is high-resolution images and detailed information provided by the museum or store, and the output is the temporarily stored data.

[1160] Step 2:

[1161] The server analyzes the digital data stored in the temporary storage directory and extracts metadata (artist name, production year, technique, etc.). The input is the stored digital data, and the output is the extracted metadata. Specific operations include using image recognition software and text analysis algorithms.

[1162] Step 3:

[1163] The server converts the analyzed digital data into an appropriate format and stores it in a database. The input is metadata and digital data, and the output is digital data stored in the database. Specifically, it performs data format conversion and indexing.

[1164] Step 4:

[1165] A user accesses the system from a terminal and requests information about a specific category, product, or artwork. The input is the user's request, and the output is the request information being sent to the server. Specific operations include receiving input through the user interface.

[1166] Step 5:

[1167] The server analyzes the user's request and searches for guide information in the database. The input is the user's request information, and the output is a list of the most suitable guides. The specific operation uses a search algorithm based on the request content.

[1168] Step 6:

[1169] The server sends a list of the most suitable guides to the user. The input is the list of guides, and the output is the guide information displayed on the user's terminal. Specifically, the data format is converted and sent.

[1170] Step 7:

[1171] The user selects the desired guide from the list and sends the information to the server. The input is the selected guide information, and the output is sending the selected information to the server. The specific operation involves operating the selection interface.

[1172] Step 8:

[1173] When the tour start time arrives, the server sends a notification of the tour start to the user and guide's terminals and loads the necessary digital data in advance. The input is the tour start time and digital data, and the output is the notification and the loaded data on the terminal. Specific operations include schedule management and data transmission.

[1174] Step 9:

[1175] The guide uses a dedicated application to operate the products and artworks while providing commentary. The input is the loaded digital data and the guide's commentary, and the output is real-time commentary and screen operation information. Specific operations include voice input and screen operation.

[1176] Step 10:

[1177] Users operate the virtual space on their devices to view artworks and products in detail. They also use the chat function to send questions. The input is the user's operations and questions, and the output is the displayed virtual space and the questions sent. Specific operations include operating the user interface and sending questions.

[1178] Step 11:

[1179] The server receives the user's question, converts it into text, and sends it to the guide. The input is the user's question, and the output is the converted text and the transmitted question. Specific operations include voice recognition software and data transmission.

[1180] Step 12:

[1181] After the tour ends, the server displays a questionnaire on the user's device and collects responses. The input is the questionnaire form and the user's responses, and the output is the collected questionnaire data. Specifically, the server displays the questionnaire form and collects responses.

[1182] Step 13:

[1183] The server analyzes the collected survey results and comments made during the tour, generates a feedback report, and provides it to museums and stores. The input is the survey results and comments, and the output is the generated feedback report. Specific operations include data analysis and report generation.

[1184] Step 14:

[1185] The server recommends related products and provides purchase links based on the artworks and products that the user shows interest in during the tour. The input is the user's interest information and related product data, and the output is the recommended products and purchase links. Specific operations include data mining and link generation.

[1186] In this way, users can enjoy the experience of viewing and purchasing high-quality artworks and merchandise without being restricted by physical constraints, and by providing customized guidance based on individual interests, a highly satisfying service is realized.

[1187] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1188] The present invention is a system that digitally archives museum exhibits and provides a virtual museum that users can access online. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, a more personalized experience is provided. Specific embodiments of each function are described below.

[1189] 1. Digital archiving of museums

[1190] server

[1191] The server receives digital data (images, videos, text, etc.) of exhibits provided by the museum and temporarily stores it. For example, when the museum uploads a high-resolution image of "Famous Painting A," the server receives this data and moves it to a temporary storage directory. The server then analyzes the digital data and extracts metadata (e.g., artist name, year of production, technique used). The server then converts the data into an appropriate format and stores it in a dedicated database. The server also indexes the data to enable quick searches and filtering later.

[1192] 2. Online tour guide matching

[1193] user

[1194] A user accesses the virtual museum from their device and requests a tour guide. For example, if they input their desire to "learn about Impressionism," this information is sent to the server. The server analyzes this request and searches for guide information in its database. For example, it creates a list of guides who are knowledgeable about "Impressionism" and sends this list to the user. The user selects the desired guide from the displayed guides and sends that information to the server. The server then requests confirmation and approval of the tour date and time from the selected guide, and if approval is granted, sends the confirmed information to the user.

[1195] 3. Conducting Virtual Tours

[1196] Server, user terminal, guide terminal

[1197] When the tour begins, the server sends a notification to the user and guide's devices that the tour is about to begin, and pre-loads the digital data necessary for the tour. The guide uses a dedicated application to operate the exhibits while giving their explanation. At this time, the guide's device sends the explanation and screen operation information to the user in real time via the server. Users can operate the virtual space on their own devices and view the exhibits in detail. Users can also send questions using the chat function. The server receives questions from users, converts them to text, and sends them to the guide. These questions are translated as needed, making multilingual support possible. The guide answers the users' questions in real time.

[1198] 4. Secondary Features of the Service

[1199] Server, user terminal

[1200] After the tour ends, the server displays a questionnaire on the user's device and collects feedback information. The collected questionnaire results and comments made during the tour are analyzed, and a feedback report is generated and provided to the museum. Based on the exhibits and artists that the user showed interest in during the tour, the server also generates information recommending products from the museum store and sends this to the user's device. Related products are displayed on the user's device, and a link to purchase them is provided. For example, related products for "Famous Painting A" are displayed, and the user can purchase them via a link to an e-commerce site.

[1201] 5. Incorporating an Emotional Engine

[1202] Emotion Engine

[1203] The emotion engine analyzes the user's facial expression data, tone of voice, and text input data collected in real time from the user's device to recognize the user's emotional state, for example, determining whether the user is excited or bored by a particular exhibit.

[1204] Use of Emotional Data

[1205] Server, guide terminal

[1206] The emotion engine analyzes the user's emotional state and sends it back to the guide terminal in real time via the server. Based on this information, the guide can dynamically adjust the content of the commentary or provide additional information to pique the user's interest. For example, if the user is not interested in a particular exhibit, the guide can introduce other related exhibits.

[1207] Feedback Improvements

[1208] server

[1209] The server generates feedback to improve the user experience based on the emotional data analyzed by the emotion engine and provides it to the museum, which can then use it to improve the exhibits and tour content.

[1210] In this way, the system of the present invention recognizes and reflects user emotions in real time, providing a more personalized and high-quality online museum experience.

[1211] The processing flow will be explained below.

[1212] 1. Digital archiving of museums

[1213] Step 1:

[1214] Server: Provides a dedicated form on the website for museum staff to upload digital data (images, videos, text, etc.) of exhibits.

[1215] Step 2:

[1216] Server: Receives uploaded digital data and moves it to a temporary storage directory. For example, if a museum uploads a high-resolution image of "Famous Painting A," the data is stored in that directory.

[1217] Step 3:

[1218] Server: Analyzes the received digital data and executes analysis scripts to extract metadata (artist name, year of production, technique, etc.). The extracted metadata is stored in a database along with the data.

[1219] Step 4:

[1220] Server: Indexes the data to enable quick searches and filtering later. The indexed data is efficiently managed in a database.

[1221] 2. Online tour guide matching

[1222] Step 1:

[1223] User: The user accesses the virtual museum website from their device, enters the tour guide request screen, and enters "I would like a tour of the Impressionists." This information is sent to the server.

[1224] Step 2:

[1225] Server: Analyzes the user's request, searches for guide information in the database, generates a list of target guides, and sends it to the user's device.

[1226] Step 3:

[1227] User: The user selects the desired guide from the displayed guides and sends the selected information to the server. For example, if the user selects "Guide B," the information is sent to the server.

[1228] Step 4:

[1229] Server: Requests confirmation and approval of the tour date and time from the selected guide, and sends the confirmation information to the user once approval is received.

[1230] 3. Conducting Virtual Tours

[1231] Step 1:

[1232] Server: When the tour start time arrives, the server sends a tour start notification to the user and guide's devices and preloads the digital data required for the tour.

[1233] Step 2:

[1234] Guide: The guide uses a dedicated application to operate the exhibits and provide commentary. Screen operations and commentary are distributed to users in real time via a server.

[1235] Step 3:

[1236] Users: Users can operate the virtual space on their own devices, view the exhibits in detail, and send questions to the guide using the chat function.

[1237] Step 4:

[1238] Server: Converts user questions into text and sends it to the guide. If necessary, translates the questions to support multiple languages.

[1239] Step 5:

[1240] Guide: The guide answers users' questions in real time.

[1241] 4. Secondary Features of the Service

[1242] Step 1:

[1243] Terminal: After the tour ends, the server displays a questionnaire on the user's terminal to collect feedback information. The user answers the questionnaire and sends the results to the server.

[1244] Step 2:

[1245] Server: Analyzes the collected survey results and comments made during the tour, generates a feedback report, and provides it to the museum.

[1246] Step 3:

[1247] Server: Generates information recommending products from the museum store based on the exhibits and artists that the user is interested in during the tour, and sends this information to the user's device.

[1248] Step 4:

[1249] Device: Display related products on the user's device and provide a purchase link. For example, related products for "Famous Painting A" are displayed, and the user can purchase them via a link to an e-commerce site.

[1250] 5. Incorporating an Emotional Engine

[1251] Step 1:

[1252] Device: Obtains facial expression data, tone of voice, text input, etc. in real time from the user's device.

[1253] Step 2:

[1254] Server: Sends the acquired data to the emotion engine to analyze the user's emotional state, for example, analyzing the emotions (excitement, indifference, etc.) when the user is viewing a particular exhibit.

[1255] Step 3:

[1256] Server: Based on the analysis results, the user's emotional state is fed back to the guide device in real time.

[1257] Step 4:

[1258] Guide: The guide dynamically adjusts its commentary based on the emotional feedback it receives, for example providing more detailed information about an exhibit that excites the user.

[1259] Step 5:

[1260] Server: After the tour ends, the emotion engine generates feedback based on the emotional data analyzed and provides it to the museum.

[1261] In this way, the specific processing content at each step ensures that users can have a high-quality experience through the virtual museum. In addition, by incorporating an emotion engine, a more personalized and engaging experience is provided.

[1262] Example 2

[1263] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1264] Conventional digital archive systems for museum exhibits simply store and display digital data of exhibits, limiting the user experience. They also lack digital tour guide recommendations and two-way communication functions, making it difficult to provide a personalized experience tailored to participants. Furthermore, they lack the ability to recognize users' emotions in real time and reflect them in the guide's commentary, making it difficult to respond flexibly to users' interests and level of understanding.

[1265] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for digitally archiving museum exhibits, means for displaying the digitized exhibits on a user terminal, means for recommending an optimal tour guide based on the user's interests and concerns, means for conducting an online tour in which the guide explains the exhibits, means for supporting two-way communication between the guide and the user in real time, means for analyzing the user's emotional data and providing feedback to the guide in real time, and means for collecting feedback from the user after the tour ends and providing the feedback to the museum. This allows the user to experience a high-quality virtual tour of the museum's exhibits through the digital archive, and receive a personalized explanation based on the real-time emotional data.

[1266] An "art museum" is a facility that displays exhibits for the general public and aims to promote education and culture.

[1267] "Exhibits" refers to cultural assets and artworks such as paintings, sculptures, crafts, and photographs displayed in art museums.

[1268] "Digital archiving" is the process of recording, preserving, and managing exhibits as digital data (images, videos, text, etc.).

[1269] "User Terminal" refers to a device, such as a computer, smartphone, or tablet, that a User uses to access the System via the Internet.

[1270] A "tour guide" is a professional or function that provides commentary and explanations about museum exhibits, either online or in person.

[1271] The "recommendation method" is part of a system that uses algorithms and filtering functions to select and suggest the most suitable tour guide based on the user's interests.

[1272] An "online tour" is a service that allows users to virtually view museum exhibits using the Internet while receiving real-time commentary from a guide.

[1273] "Two-way communication" refers to the interactive function that allows users and guides to exchange information in real time.

[1274] "Emotional data" refers to information about a user's psychological state and emotions analyzed from facial expressions, tone of voice, text input, etc.

[1275] "Feedback" refers to information such as opinions, impressions, and evaluations collected from users after the tour, and is used to improve and evaluate the service.

[1276] The present invention is a system that digitally archives museum exhibits and provides a virtual museum that users can access online. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system provides a more personalized experience. Specific embodiments of each function are described below.

[1277] Digital archiving of museum exhibits

[1278] server

[1279] The server receives digital data (images, videos, text, etc.) of exhibits provided by the museum and temporarily stores it. For example, if the museum uploads a high-resolution image of "Famous Painting A," the server receives this data and moves it to a temporary storage directory. The server analyzes the received digital data and extracts metadata (artist name, year of production, technique, etc.). Software such as image analysis algorithms is used for the analysis. The analyzed data is converted into an appropriate format and stored in a dedicated database. In addition, indexing the data allows for faster searching and filtering later.

[1280] Online tour guide matching

[1281] user

[1282] A user accesses the virtual museum from their device and requests a tour guide. For example, if they enter their request as "I would like to learn about Impressionism," this information is sent to the server. The server analyzes the user's request and searches for guide information in its database. For example, it creates a list of guides who are knowledgeable about "Impressionism" and sends this list to the user. The user selects the desired guide from the displayed guides and sends that information to the server. The server then requests confirmation and approval of the tour date and time from the selected guide, and if approval is granted, sends the confirmed information to the user.

[1283] Conducting a virtual tour

[1284] Server, user terminal, guide terminal

[1285] When the tour begins, the server sends a notification to the user's and guide's devices and pre-loads the digital data necessary for the tour. The guide uses a dedicated application to operate the exhibits while providing commentary. For example, the guide might display details of "Masterpiece A" on the screen while explaining its background and techniques. The guide's device sends the commentary and screen operation information to the user in real time via the server. Users can operate the virtual space on their own devices and view the exhibits in detail. Users can also send questions using the chat function. The server receives questions from users, converts them to text, and sends them to the guide. These questions are translated as needed, making multilingual support possible. The guide then answers the user's questions in real time.

[1286] Service Sub-Features

[1287] Server, user terminal

[1288] After the tour ends, the server displays a questionnaire on the user's device to collect feedback information. For example, feedback is collected by having the user answer the question, "How satisfied were you with this tour?" The collected feedback information is analyzed and provided to the museum. The server analyzes the collected feedback and the user's interests, generates information recommending related museum store products, and sends this information to the user's device. For example, related products for "Famous Painting A" are displayed, and the user can purchase them via a link to an e-commerce site.

[1289] Incorporating an emotion engine

[1290] Emotion engine, server

[1291] The emotion engine analyzes the user's facial expression data, tone of voice, and text input content obtained in real time from the user's device to recognize the user's emotional state. For example, it determines whether the user is excited or bored by a particular exhibit. This information is fed back to the guide device via the server. The guide then dynamically adjusts the content of the commentary based on this information. For example, if the user shows no interest in a particular exhibit, the guide will introduce other related exhibits. The server generates feedback to improve the user experience based on the emotional data analyzed by the emotion engine and provides it to the museum.

[1292] Examples of concrete examples and prompts

[1293] For example, if an art museum were to digitally archive an "Impressionist Exhibition," it would follow the steps below.

[1294] 1. The museum uploads a high-resolution image of "Masterpiece A" to the server.

[1295] 2. The server analyzes the image data and extracts metadata such as the artist's name and technique.

[1296] 3. The user requests, "I want to learn about Impressionism," and the server matches them with a suitable tour guide.

[1297] 4. When the tour start time arrives, the server notifies the user and guide devices and the virtual tour begins.

[1298] 5. The guide explains the details of the exhibits, and when the user asks questions, they are sent to the guide via the server.

[1299] 6. After the tour, a survey will be displayed on the user's device to collect feedback information.

[1300] Prompt Sentence Examples

[1301] I want to book a virtual tour about Impressionism.

[1302] I want to know more about Masterpiece A.

[1303] The above is an embodiment of the present invention, which allows users to experience a high-quality virtual tour and receive personalized commentary in real time.

[1304] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1305] Step 1:

[1306] The server receives the digital data from the museum.

[1307] Input: High-resolution images, videos, and text data uploaded by museums.

[1308] Data processing: The input data is saved in a temporary directory.

[1309] Output: Digital data saved in temporary directory.

[1310] Specific operation: When the museum uploads a high-resolution image of "Famous Painting A," the server receives this data and moves it to a temporary storage directory.

[1311] Step 2:

[1312] The server analyzes the received digital data and extracts the metadata.

[1313] Input: Digital data stored in the temporary directory.

[1314] Data processing: Analyze the digital data using software such as image analysis algorithms.

[1315] Output: Extracted metadata (artist name, year of production, technique, etc.).

[1316] Specific operation: The server analyzes the stored image data and automatically extracts information about the artist and era of the work.

[1317] Step 3:

[1318] The server converts the parsed data into an appropriate format and stores it in a database.

[1319] Input: Extracted metadata and original digital data.

[1320] Data processing: Image data is converted into a specific compressed format and stored in a database along with metadata.

[1321] Output: Digital data and metadata stored in a database.

[1322] Specific operation: The server converts the analyzed data into an appropriate format and stores the digital data and metadata in a database.

[1323] Step 4:

[1324] Users access the virtual museum from their own devices and request a tour guide.

[1325] Input: A tour guide request sent from the user's device (e.g., "I want to learn about Impressionism").

[1326] Data processing: The server receives and analyzes the user's request.

[1327] Output: The tour guide request sent to the server.

[1328] What it does: A user types in "I want to learn about Impressionism," and this information is sent to the server.

[1329] Step 5:

[1330] The server analyzes the tour guide request and matches the most suitable guide.

[1331] Input: User request data.

[1332] Data processing: The server searches the guide information in the database and lists related guides.

[1333] Output: A list of guides that will be sent to the user.

[1334] What it does: The server compiles a list of detailed guides to "Impressionism" from its database and sends this list to the user.

[1335] Step 6:

[1336] The user selects the desired guide and submits it to the server, which requests confirmation and approval of the tour date and time.

[1337] Input: User selected guide information.

[1338] Data processing: The server confirms the tour date and time with the selected guide and requests approval.

[1339] Output: Tour information approved by the guide.

[1340] Specific operation: The user selects the desired guide and sends the information to the server. The server requests the guide to confirm and approve the tour date and time, and once approval is received, the confirmation information is sent to the user.

[1341] Step 7:

[1342] At the start time of the tour, the server sends a notification and pre-loads the digital data required for the tour.

[1343] Input: Confirmed tour information.

[1344] Data processing: Preload the necessary digital data.

[1345] Output: Tour start notification to user and guide devices.

[1346] Specific operation: When the tour start time arrives, the server sends a notification and loads the digital data required for the tour into the user's and guide's devices in advance.

[1347] Step 8:

[1348] The guide will provide commentary while operating the exhibits, and users can send questions via their terminal.

[1349] Input: Data about the exhibits operated by the guide, user question data.

[1350] Data processing: The server sends the commentary and screen operation information to the user, and sends the user's questions to the guide.

[1351] Output: Instructions sent to the user, user questions sent to the guide.

[1352] Specific operation: The guide explains "Famous Painting A," and when the user asks a question, the server sends the question to the guide, who then answers in real time.

[1353] Step 9:

[1354] The emotion engine analyzes the user's emotional data and provides feedback to the guide via the server.

[1355] Input: User facial expression data, tone of voice, and text input.

[1356] Data processing: The emotion engine analyzes the emotion data and provides feedback to the guide via the server.

[1357] Output: The user's emotional state, which is fed back to the guide.

[1358] Specific operation: The emotion engine analyzes the user's emotions and provides the guide with the user's excitement level or boredom level in real time.

[1359] Step 10:

[1360] After the tour ends, the server displays a questionnaire on the user's device to collect feedback information.

[1361] Input: User survey data after the tour.

[1362] Data processing: Analyze the collected survey results and generate a feedback report to provide to the museum.

[1363] Output: Feedback report provided to the museum, related product recommendations displayed on the user's device.

[1364] Specific operation: After the tour ends, a questionnaire is displayed on the user's device, and feedback information is collected and analyzed. The analysis results are provided to the museum, and related product recommendations are displayed on the user's device.

[1365] (Application example 2)

[1366] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1367] Conventional online virtual tour systems for museums and stores have struggled to provide a personalized experience based on the user's interests and emotions. Furthermore, they lacked the functionality to analyze the user's facial expressions and tone of voice to provide real-time product recommendations and information. This resulted in low user satisfaction.

[1368] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for digitalizing exhibits, means for displaying the digitalized exhibits on a user terminal, means for recommending an optimal guide based on the user's interests and concerns, means for conducting an online guided tour in which the guide explains the exhibits, means for supporting two-way communication between the guide and the user in real time, means for analyzing the user's facial expression data and voice data in real time to recognize the user's emotional state, and means for recommending products that will interest the user and providing additional information based on the user's emotional state. This enables a personalized and high-quality online experience that is tailored to the user's emotions and interests.

[1369] "Means of digitalizing exhibits" refers to the technology used to convert physical exhibits into digital data and store it in a database.

[1370] "Means for displaying digitalized exhibits on user terminals" refers to technology for displaying exhibits stored as digital data on terminals used by users.

[1371] "Means for recommending the most suitable guide based on the user's interests and concerns" refers to technology that analyzes the interests and concerns expressed by the user and automatically selects and recommends the guide that best suits them.

[1372] "Means for conducting online guided tours in which a guide explains exhibits" refers to technology that enables a guide to provide explanations about digitized exhibits and share the content with users in real time.

[1373] "Means for supporting two-way communication between a guide and a user in real time" refers to technology that enables a guide and a user to have two-way communication in real time.

[1374] "Means for analyzing a user's facial expression data and voice data in real time to recognize their emotional state" refers to technology that uses a camera or microphone to capture the user's facial expressions and voice, analyzes that data, and recognizes the user's emotional state in real time.

[1375] The "means for recommending products and providing additional information that may interest a user based on the user's emotional state" is a technology that dynamically provides products and related information that may be of interest to a user based on the user's recognized emotional state.

[1376] This invention is a system that allows users to experience exhibits and products in a virtual space, and provides personalized information by recognizing the user's emotions in real time. This system is constructed mainly using a server, a user terminal, and a guide terminal, and specific embodiments are described below.

[1377] Hardware and Software Use

[1378] The server is installed on the cloud and performs temporary data storage, analysis, and database management. Smartphones, tablets, PCs, etc. are used as user and guide devices, and these devices must be equipped with a camera and microphone. The main software used is as follows:

[1379] Google Cloud Vision API: A service for emotion analysis that analyzes image data and recognizes the user's facial expressions.

[1380] Firebase: A database service that processes and stores data in real time, managing user sentiment data and feedback information.

[1381] WebRTC: A technology that enables real-time communication, used for video chat and data streaming between users and guides.

[1382] Data acquisition and analysis

[1383] The server receives and temporarily stores digital data (images, videos, text, etc.) of exhibits and products provided by museums and stores. When high-resolution image data and detailed information are uploaded, the server analyzes it, extracts appropriate metadata (e.g., product name, price, description), and stores it in a database.

[1384] User information analysis and matching

[1385] When a user visits the virtual store, the user's device captures facial expression data using a camera and voice data using a microphone. This data is sent in real time to the Google Cloud Vision API, which analyzes the user's emotional state. The analysis results are stored in Firebase and fed back to the guide via the server.

[1386] Online tours and guided tours

[1387] When the tour begins, the server sends a notification to the user's and guide's devices and pre-loads the digital data necessary for the tour. The guide uses a dedicated application to interact with the exhibits and products while providing commentary, which is then sent to the user's device in real time. The user can move their viewpoint in any direction within the virtual space to view the exhibits in detail. They can also use the chat function to send questions to the guide.

[1388] Feedback and recommendation system

[1389] After the tour ends, the server displays a questionnaire screen on the user's device to collect feedback. Based on this feedback information and emotional data obtained during the tour, the server generates recommendation information for products and exhibits that the user showed interest in and provides it to the user's device. This information also includes links to related products and services, allowing users to smoothly make purchases or check details.

[1390] Examples of concrete examples and prompts

[1391] As a concrete example, if a user is exploring the wearable devices section of a virtual store and their smartphone camera captures their facial expression and detects "excitement," the system can analyze this data and automatically display more information and reviews related to the product they are particularly interested in (e.g., the latest smartwatch).

[1392] Example prompt sentence:

[1393] "Analyze users' facial expressions and tone of voice in real time and make appropriate product recommendations based on their emotional state (e.g., joy, anger, sadness, happiness, etc.). Use Firebase and Google Cloud Vision API to store user emotion data and notify the shopping guide in real time."

[1394] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1395] Step 1:

[1396] The server receives and temporarily stores digital data (e.g., images, videos, text) of exhibits and products provided by museums and stores. It then analyzes this data to extract metadata (e.g., product name, price, description) and stores it in a database. The input is digital data, and the output is metadata.

[1397] Step 2:

[1398] The user device captures the user's facial expression data using a camera and voice data using a microphone. This data is sent in real time to the Google Cloud Vision API, which analyzes the user's emotional state. The input is the user's facial expression and voice data, and the output is the analysis result of the emotional state.

[1399] Step 3:

[1400] The server stores the emotional state data obtained as a result of the analysis in Firebase, and uses this data to generate recommendation data for products that will interest the user. The input is the emotional state analysis results, and the output is recommendation data.

[1401] Step 4:

[1402] When a user accesses a virtual store, the server recommends the most suitable guide based on the user's interests. Based on input data about interests obtained from the user's device, the server searches for guide information in the database and lists the most suitable guides. The input is data about the user's interests, and the output is a list of guides.

[1403] Step 5:

[1404] At the start of the tour, the server sends notifications to the user and guide's devices and preloads digital data about the virtual store. The user and guide start a real-time video chat using WebRTC, and the server relays the communication between them. The input is a connection request from both devices, and the output is the establishment of the connection.

[1405] Step 6:

[1406] During the tour, the guide explains the exhibits and products, and the content is sent to the user in real time via the server. Users can use the chat function to send questions, which the server converts into text and sends to the guide. The input is the guide's explanation data and the user's question data, and the output is the textual question data and explanation content.

[1407] Step 7:

[1408] After the tour ends, the server displays a questionnaire screen on the user's device to collect feedback. The collected feedback information and emotional data collected during the tour are analyzed, and recommendation information is generated for products and exhibits that the user showed interest in. The input is feedback data and emotional data, and the output is recommendation information.

[1409] Step 8:

[1410] The server sends the final recommendation information to the user's device and provides detailed information and a purchase link for related products and services, allowing the user to smoothly purchase products or check details. The input is the generated recommendation information, and the output is the detailed information and purchase link displayed on the user's device.

[1411] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1412] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1413] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1414] [Fourth embodiment]

[1415] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1416] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1417] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1418] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1419] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1420] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1421] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1422] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1423] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1424] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1425] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1426] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1427] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1428] The present invention provides a system that digitally archives art museum exhibits and provides a virtual museum that users can access online. Specific embodiments of each function are described below.

[1429] 1. Digital archiving of museums

[1430] server

[1431] The server receives digital data (images, videos, text, etc.) of exhibits provided by the museum and temporarily stores it. For example, when the museum uploads a high-resolution image of "Famous Painting A," the server receives this data and stores it in a temporary storage directory. The server then analyzes the digital data and extracts metadata (e.g., artist name, year of production, technique used). The server then converts the data into an appropriate format and stores it in a dedicated database. The server also indexes the data to enable quick searches and filtering later.

[1432] 2. Online tour guide matching

[1433] user

[1434] A user accesses the virtual museum from their device and requests a tour guide. For example, if they input their desire to "learn about Impressionism," this information is sent to the server. The server analyzes this request and searches for guide information in its database. For example, it creates a list of guides who are knowledgeable about "Impressionism" and sends this list to the user. The user selects the desired guide from the displayed guides and sends that information to the server. The server then requests confirmation and approval of the tour date and time from the selected guide, and if approval is granted, sends the confirmed information to the user.

[1435] 3. Conducting Virtual Tours

[1436] Server, user terminal, guide terminal

[1437] When the tour begins, the server sends a notification to the user and guide's devices that the tour is about to begin, and pre-loads the digital data necessary for the tour. The guide uses a dedicated application to operate the exhibits while giving their explanation. At this time, the guide's device sends the explanation and screen operation information to the user in real time via the server. Users can operate the virtual space on their own devices and view the exhibits in detail. Users can also send questions using the chat function. The server receives questions from users, converts them to text, and sends them to the guide. These questions are translated as needed, making multilingual support possible. The guide answers the users' questions in real time.

[1438] 4. Secondary Features of the Service

[1439] Server, user terminal

[1440] After the tour ends, the server displays a survey on the user's device and collects responses. The collected survey results and comments made during the tour are analyzed, and a feedback report is generated and provided to the museum. This allows the museum to improve its services. Furthermore, based on the exhibits and artists that the user showed interest in during the tour, the server recommends products from the museum store and provides a purchasing link to the user's device. For example, if a user is interested in a replica of "Masterpiece A," related products will be displayed and a link to purchase them will be provided.

[1441] This will provide museums with new revenue streams and allow visitors to have a high-quality guided museum experience remotely.

[1442] The processing flow will be explained below.

[1443] 1. Digital archiving of museums

[1444] Step 1:

[1445] Server: Provides a dedicated upload form to receive digital data (images, videos, text, etc.) of exhibits provided by the museum. When a museum staff member uploads a high-resolution image of "Masterpiece A," the server receives this data and moves it to a directory for temporary storage.

[1446] Step 2:

[1447] Server: Analyzes the received digital data and runs analysis scripts to automatically extract metadata (artist name, year of production, technique, etc.) Once the metadata is extracted, the server stores this data in a digital archive database.

[1448] Step 3:

[1449] Server: Indexes the data to enable quick searches and filtering later. The indexed data is efficiently managed in a database.

[1450] 2. Online tour guide matching

[1451] Step 1:

[1452] Device: The user accesses the virtual museum website from their device, enters the tour guide request screen, and enters "I would like a tour of the Impressionists." This information is sent to the server.

[1453] Step 2:

[1454] Server: Analyzes the user's request, searches for guide information in the database, generates a list of target guides, and sends it to the user's device.

[1455] Step 3:

[1456] Terminal: The user selects the desired guide from the displayed guides and sends the selected information to the server. For example, if "Guide B" is selected, the information is sent to the server.

[1457] Step 4:

[1458] Server: Requests confirmation and approval of the tour date and time from the selected guide, and sends the confirmation information to the user once approval is received.

[1459] 3. Conducting Virtual Tours

[1460] Step 1:

[1461] Server: When the tour start time arrives, the server sends a tour start notification to the user and guide's devices and preloads the digital data required for the tour.

[1462] Step 2:

[1463] Guide: The guide uses a dedicated application to operate the exhibits and provide commentary. Screen operations and commentary are distributed to users in real time via a server.

[1464] Step 3:

[1465] Users: Users can operate the virtual space on their own devices, view the exhibits in detail, and send questions to the guide using the chat function.

[1466] Step 4:

[1467] Server: Converts user questions into text and sends it to the guide. If necessary, translates the questions to support multiple languages.

[1468] Step 5:

[1469] Guide: The guide answers users' questions in real time.

[1470] 4. Secondary Features of the Service

[1471] Step 1:

[1472] Terminal: After the tour ends, the server displays a questionnaire on the user's terminal to collect feedback information. The user answers the questionnaire and sends the results to the server.

[1473] Step 2:

[1474] Server: Analyzes the collected survey results and comments made during the tour, generates a feedback report, and provides it to the museum.

[1475] Step 3:

[1476] Server: Generates information recommending products from the museum store based on the exhibits and artists that the user is interested in during the tour, and sends this information to the user's device.

[1477] Step 4:

[1478] Device: Display related products on the user's device and provide a purchase link. For example, related products for "Famous Painting A" are displayed, and the user can purchase them via a link to an e-commerce site.

[1479] In this way, through the specific processing content at each step, users can have a high-quality experience through the virtual museum.

[1480] Example 1

[1481] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1482] There is a need for a system that allows general visitors to easily and effectively view cultural assets in museums. However, existing systems have issues such as insufficient digital archiving and the inability to communicate in real time between users and guides. Furthermore, they lack the ability to provide a personalized experience based on users' interests, which leads to a decline in user satisfaction.

[1483] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1484] In this invention, the server includes means for digitally archiving the museum's cultural assets, means for displaying the digitized cultural assets on an information processing terminal, and means for recommending the most appropriate tour guide based on the user's interests. This enables digital archiving of cultural assets and personalized online viewing based on the archived data. The server also includes means for extracting metadata from received digital data, converting it into a storage format, and storing it in a database; means for analyzing the user's viewing request and searching for tour guide information in the database; means for requesting the tour guide to confirm and approve the date and time of the viewing; means for sending a tour start notification and preloading the necessary digital data; and means for receiving user operations and questions during the viewing of the cultural assets, converting them into text, translating them into multiple languages, and sending them to the tour guide. This enables real-time two-way communication between the user and the tour guide, providing a richer online viewing experience.

[1485] An "art museum" is a facility that collects, preserves, exhibits, and makes available to the public cultural assets and works of art.

[1486] "Cultural properties" refer to items, documents, and exhibits that have historical, cultural, or academic value.

[1487] "Digital archiving" is the process of preserving physical cultural assets as digital data to increase their convenience.

[1488] "Information processing terminal" refers to a device such as a computer, tablet, or smartphone that displays and manipulates digital data.

[1489] "User" means an individual or group who uses the system to view digitally archived cultural property.

[1490] A "facilitator" is an expert or guide who provides explanations about cultural properties to users on the system.

[1491] "Online viewing" is the experience of viewing digitally archived cultural assets via the Internet.

[1492] "Two-way communication" is a function that allows users and moderators to exchange messages and information in real time.

[1493] "Metadata" is additional information about digital data, such as the artist's name, the year of production, and the technique used.

[1494] A "database" is a system for efficiently storing, managing, and searching multiple digital data.

[1495] "Multilingual translation" is a function that translates into different languages ​​and supports communication between users and facilitators who speak different languages.

[1496] The present invention is a system that digitally archives cultural assets in art museums and provides a virtual museum that users can access online. The system of the present invention operates primarily on a server and functions in conjunction with information processing terminals (user terminals and guide terminals). Specific embodiments of each function are described below.

[1497] server

[1498] The server receives and temporarily stores digital data (images, videos, text, etc.) of cultural assets provided by the museum. At this time, the server analyzes the digital data and uses image processing algorithms and natural language processing technology to extract metadata (artist name, year of production, technique, etc.). The server then converts the digital data into an appropriate storage format and stores it in a dedicated database. The server also indexes the data, making it quickly searchable.

[1499] For example, if a museum provides a high-resolution image of "Famous Painting A," the server stores the image in a temporary directory and uses image processing algorithms to extract the artist's name and the year of production. The server then converts the image along with the extracted metadata into an appropriate format and stores it in the database.

[1500] user

[1501] A user accesses the virtual museum from their device and requests a tour guide. For example, if the user inputs their desire to "learn about Impressionism," this information is sent to the server. The server analyzes this request and searches for guide information in its database. The search results list guides who are knowledgeable about Impressionism and provides this list to the user. The user selects the desired guide and requests confirmation and approval of the tour date and time from the server.

[1502] Specifically, when a user requests a "tour guide of contemporary art," the server searches the database for guides knowledgeable about contemporary art and presents them to the user. The user selects the desired guide and sends the information to the server.

[1503] Guide terminal

[1504] When the tour begins, the server sends a notification to the user's and guide's devices. The server also pre-loads the digital data necessary for the tour. The guide uses a dedicated application to operate the exhibits while providing commentary on the cultural properties. The content of this commentary and screen operation information are sent to the user in real time via the server.

[1505] For example, while the guide is explaining "Famous Painting A," the user may ask a question via chat, such as, "I'd like to know about the techniques used in this painting." The question is sent to the guide via the server and translated as necessary. The guide can then answer the question in real time.

[1506] Server and user terminal

[1507] After the tour ends, the server displays a questionnaire on the user's device and collects their responses. The server then analyzes the collected questionnaire results and comments made during the tour to generate a feedback report and provide it to the museum, providing specific data for the museum to use to improve its services. Furthermore, based on the cultural assets the user showed interest in during the tour, the server recommends related museum store products to the user's device and provides a link to purchase them.

[1508] For example, if a user is interested in a replica of "Famous Painting A," the system will display related products and provide a link to purchase them, enabling museums to secure new revenue streams and providing visitors with a high-quality, guided museum experience from a remote location.

[1509] Example prompts for generative AI models

[1510] "Please detail the steps a user can take to request an Impressionist tour guide in your virtual museum system."

[1511] "Please explain in detail how the server receives and stores data from the museum in the digital archive system."

[1512] "Please detail the steps your server takes to display and collect a survey from users after the tour has finished."

[1513] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1514] Step 1: Receiving and temporarily storing digital data

[1515] The server receives digital data of cultural assets (images, videos, text, etc.) from the museum and stores the received data in a temporary directory.

[1516] Input: Digital data provided by the museum

[1517] Output: Digital data stored in a temporary directory

[1518] Specifically, when the museum uploads a high-resolution image file of "Masterpiece A," the server temporarily stores this data in a storage directory.

[1519] Step 2: Extracting Metadata

[1520] The server analyzes the digital data stored in the temporary storage directory and extracts metadata, such as the artist's name, production year, and technique, using image processing algorithms and natural language processing technology.

[1521] Input: Digital data stored in a temporary directory

[1522] Output: Extracted metadata

[1523] Specifically, the server automatically recognizes the artist's name, year of production, and techniques used from the image of "Masterpiece A" and extracts them as metadata.

[1524] Step 3: Convert and save the data format

[1525] The server converts the digital data along with the extracted metadata into a designated storage format, and stores the converted data in a dedicated database.

[1526] Input: Digital data, extracted metadata

[1527] Output: Data converted to a suitable storage format, data stored in a database

[1528] Specifically, it converts large image files into a format suitable for web display and stores the converted data and metadata in a database.

[1529] Step 4: Indexing

[1530] The server indexes the stored data, which allows for fast searches.

[1531] Input: Data stored in a database

[1532] Output: Indexed database

[1533] Specifically, the server adds metadata such as the artist's name and technique to the index, making it easy to search later.

[1534] Step 5: Request a tour guide

[1535] Users access the virtual museum from their own devices and request the tour guide of their choice, for example, "I want to learn about Impressionism," and send the request to the server.

[1536] Input: User's tour guide request

[1537] Output: Request sent to the server

[1538] Specifically, the user requests a "contemporary art tour guide" and the server receives the request, "I would like to learn about Impressionism."

[1539] Step 6: Request analysis and guided search

[1540] The server analyzes the user's request, searches for the moderator's information in the database, and lists detailed guides to "Impressionism" and provides them to the user.

[1541] Input: User request

[1542] Output: Guide list as search results

[1543] Specifically, the server searches a database for detailed guides on "Impressionism," lists them, and provides them to the user.

[1544] Step 7: View and select guide list

[1545] The user browses the list of guides sent from the server, selects the guide he or she wants, and sends the selected information to the server.

[1546] Input: Guide list from the server

[1547] Output: Selected guide information

[1548] Specifically, the user selects the desired guide from the displayed guides and sends that information to the server.

[1549] Step 8: Review and approve your guide

[1550] The server requests the selected guide to confirm and approve the viewing date and time. If the guide approves, the information is sent to the user to confirm the viewing details.

[1551] Input: User-selected guide information

[1552] Output: Guide approval status, notification to user

[1553] Specifically, the server requests the selected guide to confirm the viewing date and time, and notifies the user after approval.

[1554] Step 9: Notification of viewing start and data loading

[1555] When the viewing start time arrives, the server sends a notification to the user and guide's devices, and also preloads the digital data necessary for the viewing.

[1556] Input: Viewing start time

[1557] Output: Viewing start notification, loaded digital data

[1558] Specifically, the server sends a notification that viewing has begun and loads exhibit information.

[1559] Step 10: Guide operation and explanation

[1560] The guide terminal uses a dedicated application to operate the exhibits while providing commentary, and transmits the commentary and screen operation information to the user in real time.

[1561] Input: Exhibit operated by the guide

[1562] Output: Real-time commentary and control information

[1563] Specifically, the guide operates "Masterpiece A" while providing commentary, and transmits this information to the user in real time.

[1564] Step 11: User Actions and Questions

[1565] The user device allows them to view the virtual space in detail and send questions using the chat function.

[1566] Input: User question

[1567] Output: The question sent to the server

[1568] Specifically, when a user submits a question such as "What is this technique?", the question is sent to the server.

[1569] Step 12: Receiving and answering questions

[1570] The server converts user questions into text, translates them if necessary, and sends them to the guide, who then answers the questions in real time.

[1571] Input: User question

[1572] Output: The textual questions sent to the guide

[1573] Specifically, the server receives the user's question, translates it, and sends it to the guide, who then answers it.

[1574] Step 13: Display the survey and collect responses

[1575] After viewing, the server displays a questionnaire on the user's device and collects responses.

[1576] Input: Survey display request after viewing ends

[1577] Output: Collected survey responses

[1578] Specifically, the server displays a questionnaire on the user's device asking, "What did you think of the tour?" and collects the user's responses.

[1579] Step 14: Analyze and provide feedback

[1580] The server analyzes the collected survey results and comments made during the tour, generates a feedback report, and provides it to the museum.

[1581] Input: Collected survey responses and comments made during the tour

[1582] Output: Feedback report

[1583] Specifically, the server generates a feedback report based on the analysis results and provides it to the museum.

[1584] Step 15: Recommend a product and provide a link to purchase

[1585] The server recommends related museum store products based on the exhibits and artists the user has shown interest in during the tour and provides a purchase link to the user's device.

[1586] Input: Exhibits and artists that the user is interested in

[1587] Output: Recommended products and purchase links

[1588] Specifically, if a user is interested in a replica of "Famous Painting A," related products will be displayed and a purchase link will be provided.

[1589] (Application example 1)

[1590] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1591] In traditional art museums and stores, due to physical distance and time constraints, visitors had to travel to the actual site to view and purchase artworks in detail. Furthermore, due to a lack of guidance tailored to the individual interests of visitors, it was difficult to provide a satisfying experience.

[1592] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1593] In this invention, the server includes means for archiving digital data of artworks, means for displaying the digitized artworks on a terminal device, means for recommending the most suitable guide based on the user's interests, means for conducting an online tour in which the guide explains the artworks, means for supporting real-time two-way communication between the guide and the user, means for archiving digital data of store products, means for recommending the most suitable product guide based on the user's interests, and means for conducting an online tour in which the guide explains and suggests details about the products. This allows visitors to view and purchase artworks and products in detail without being restricted by physical distance or time. Furthermore, by providing guidance tailored to the user's individual interests, a highly satisfying experience can be achieved.

[1594] "Artworks" refers to artworks such as paintings, sculptures, and crafts that are exhibited in museums and galleries.

[1595] "Digital data" refers to information such as images, videos, and text recorded in electronic form.

[1596] "Archiving" refers to the process of storing information and data for a long period of time, organizing and storing it so that it can be easily searched and used later.

[1597] "End-device" refers to a terminal device that is directly operated by the user, and includes smartphones, tablets, PCs, smart glasses, etc.

[1598] A "guide" is someone with the expertise to provide users with detailed information about artworks and products and conduct online tours.

[1599] "Recommendation" means selecting the most suitable guide or product based on the user's interests and proposing it to the user.

[1600] An "online tour" is a virtual tour that takes place in real time over the Internet.

[1601] "Two-way communication" refers to a form of communication in which both parties (guide and user) can exchange information with each other.

[1602] "Products" refers to all items sold in stores, including fashion items, home appliances, stationery, etc.

[1603] "Details" are detailed descriptions or information about a particular subject.

[1604] "Proposal" means introducing solutions or products that meet the user's needs.

[1605] "Store" refers to a physical location or online platform for offering goods or services.

[1606] This invention is a system that digitally archives artworks and store merchandise and provides online guides and tours to users. Specific embodiments of each function will be described below.

[1607] 1. Archiving digital data

[1608] The server receives and temporarily stores product data (images, videos, text, etc.) provided by art objects or stores. When museums or stores upload high-resolution images of specific art objects or products, the server receives this data and stores it in a temporary storage directory. The server then analyzes the digital data and extracts metadata (e.g., artist name, year of creation, technique). This data is converted into an appropriate format and stored in a dedicated database. Indexing allows for quicker searching and filtering later.

[1609] 2. Recommendations based on user interests

[1610] Users access the system from their own devices (smartphones, tablets, smart glasses, etc.) and request guidance on a specific category, product, or artwork. For example, if a user enters a request such as "I'd like to see the latest winter fashion items," this information is sent to the server. The server analyzes this request and searches for guide information in its database. It then creates a list of the most suitable guides and sends this list to the user. The user then selects the desired guide from the displayed list and sends the information to the server.

[1611] 3. Online tours

[1612] When the tour begins, the server sends a notification to the user and guide's devices that the tour is about to begin, and pre-loads the digital data necessary for the tour. The guide uses a dedicated application to operate the products and artworks while providing commentary. At this time, the guide's device sends the commentary and screen operation information to the user in real time via the server. Users can operate the virtual space on their own devices to view the artworks and products in detail. Users can also send questions using the chat function. The server receives questions from users, converts them to text, and sends them to the guide. These questions are translated as needed, making multilingual support possible. The guide answers the questions received from users in real time.

[1613] 4. Secondary Features of the Service

[1614] After the tour ends, the server displays a questionnaire on the user's device and collects their responses. The collected questionnaire results and comments made during the tour are analyzed, and a feedback report is generated and provided to the museum or store. This allows for service improvements. Furthermore, based on the art pieces and products that the user showed interest in during the tour, related products are recommended and a purchasing link is provided to the user's device. For example, if a user is interested in the latest winter fashion items, related products are displayed and a link to purchase them is provided. This provides museums and stores with a new source of revenue and allows users to enjoy a high-quality guided experience from a remote location.

[1615] Adding specific examples

[1616] For example, if a user requests to "see the latest winter fashion items," the server will recommend the most suitable guide, who will then introduce "latest winter fashion item A" and explain its features and materials. Also, if a user asks a question about a product during the tour, the server will convert the question into text and transmit it to the guide in real time, providing an environment where an answer can be obtained immediately.

[1617] Prompt Sentence Examples

[1618] User request: "I want to see the latest winter fashion items"

[1619] Guide's answer: "Here's the latest winter fashion item, item A. It's made by a famous designer and is made from high-quality cashmere and wool. Its unique style and comfort make it a popular item in 2023."

[1620] This invention allows users to enjoy the experience of appreciating and purchasing high-quality artworks and merchandise without being physically restricted, and also provides customized guidance based on the user's individual interests, resulting in a highly satisfying service.

[1621] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1622] Step 1:

[1623] The server receives product data (images, videos, text, etc.) provided by the art works or stores and stores them in a temporary directory. The input is high-resolution images and detailed information provided by the museum or store, and the output is the temporarily stored data.

[1624] Step 2:

[1625] The server analyzes the digital data stored in the temporary storage directory and extracts metadata (artist name, production year, technique, etc.). The input is the stored digital data, and the output is the extracted metadata. Specific operations include using image recognition software and text analysis algorithms.

[1626] Step 3:

[1627] The server converts the analyzed digital data into an appropriate format and stores it in a database. The input is metadata and digital data, and the output is digital data stored in the database. Specifically, it performs data format conversion and indexing.

[1628] Step 4:

[1629] A user accesses the system from a terminal and requests information about a specific category, product, or artwork. The input is the user's request, and the output is the request information being sent to the server. Specific operations include receiving input through the user interface.

[1630] Step 5:

[1631] The server analyzes the user's request and searches for guide information in the database. The input is the user's request information, and the output is a list of the most suitable guides. The specific operation uses a search algorithm based on the request content.

[1632] Step 6:

[1633] The server sends a list of the most suitable guides to the user. The input is the list of guides, and the output is the guide information displayed on the user's terminal. Specific operations include data format conversion and transmission.

[1634] Step 7:

[1635] The user selects the desired guide from the list and sends the information to the server. The input is the selected guide information, and the output is sending the selected information to the server. The specific operation involves operating the selection interface.

[1636] Step 8:

[1637] When the tour start time arrives, the server sends a notification of the tour start to the user and guide's terminals and loads the necessary digital data in advance. The input is the tour start time and digital data, and the output is the notification and the loaded data on the terminal. Specific operations include schedule management and data transmission.

[1638] Step 9:

[1639] The guide uses a dedicated application to operate the products and artworks while providing commentary. The input is the loaded digital data and the guide's commentary, and the output is real-time commentary and screen operation information. Specific operations include voice input and screen operation.

[1640] Step 10:

[1641] Users operate the virtual space on their devices to view artworks and products in detail. They also use the chat function to send questions. The input is the user's operations and questions, and the output is the displayed virtual space and the questions sent. Specific operations include operating the user interface and sending questions.

[1642] Step 11:

[1643] The server receives the user's question, converts it into text, and sends it to the guide. The input is the user's question, and the output is the converted text and the transmitted question. Specific operations include voice recognition software and data transmission.

[1644] Step 12:

[1645] After the tour ends, the server displays a questionnaire on the user's device and collects responses. The input is the questionnaire form and the user's responses, and the output is the collected questionnaire data. Specifically, the server displays the questionnaire form and collects responses.

[1646] Step 13:

[1647] The server analyzes the collected survey results and comments made during the tour, generates a feedback report, and provides it to museums and stores. The input is the survey results and comments, and the output is the generated feedback report. Specific operations include data analysis and report generation.

[1648] Step 14:

[1649] The server recommends related products and provides purchase links based on the artworks and products that the user shows interest in during the tour. The input is the user's interest information and related product data, and the output is the recommended products and purchase links. Specific operations include data mining and link generation.

[1650] In this way, users can enjoy the experience of viewing and purchasing high-quality artworks and merchandise without being restricted by physical constraints, and by providing customized guidance based on individual interests, a highly satisfying service is realized.

[1651] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1652] The present invention is a system that digitally archives museum exhibits and provides a virtual museum that users can access online. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, a more personalized experience is provided. Specific embodiments of each function are described below.

[1653] 1. Digital archiving of museums

[1654] server

[1655] The server receives digital data (images, videos, text, etc.) of exhibits provided by the museum and temporarily stores it. For example, when the museum uploads a high-resolution image of "Famous Painting A," the server receives this data and moves it to a temporary storage directory. The server then analyzes the digital data and extracts metadata (e.g., artist name, year of production, technique used). The server then converts the data into an appropriate format and stores it in a dedicated database. The server also indexes the data to enable quick searches and filtering later.

[1656] 2. Online tour guide matching

[1657] user

[1658] A user accesses the virtual museum from their device and requests a tour guide. For example, if they input their desire to "learn about Impressionism," this information is sent to the server. The server analyzes this request and searches for guide information in its database. For example, it creates a list of guides who are knowledgeable about "Impressionism" and sends this list to the user. The user selects the desired guide from the displayed guides and sends that information to the server. The server then requests confirmation and approval of the tour date and time from the selected guide, and if approval is granted, sends the confirmed information to the user.

[1659] 3. Conducting Virtual Tours

[1660] Server, user terminal, guide terminal

[1661] When the tour begins, the server sends a notification to the user and guide's devices that the tour is about to begin, and pre-loads the digital data necessary for the tour. The guide uses a dedicated application to operate the exhibits while giving their explanation. At this time, the guide's device sends the explanation and screen operation information to the user in real time via the server. Users can operate the virtual space on their own devices and view the exhibits in detail. Users can also send questions using the chat function. The server receives questions from users, converts them to text, and sends them to the guide. These questions are translated as needed, making multilingual support possible. The guide answers the users' questions in real time.

[1662] 4. Secondary Features of the Service

[1663] Server, user terminal

[1664] After the tour ends, the server displays a questionnaire on the user's device and collects feedback information. The collected questionnaire results and comments made during the tour are analyzed, and a feedback report is generated and provided to the museum. Based on the exhibits and artists that the user showed interest in during the tour, the server also generates information recommending products from the museum store and sends this to the user's device. Related products are displayed on the user's device, and a link to purchase them is provided. For example, related products for "Famous Painting A" are displayed, and the user can purchase them via a link to an e-commerce site.

[1665] 5. Incorporating an Emotional Engine

[1666] Emotion Engine

[1667] The emotion engine analyzes the user's facial expression data, tone of voice, and text input data collected in real time from the user's device to recognize the user's emotional state, for example, determining whether the user is excited or bored by a particular exhibit.

[1668] Use of Emotional Data

[1669] Server, guide terminal

[1670] The emotion engine analyzes the user's emotional state and sends it back to the guide terminal in real time via the server. Based on this information, the guide can dynamically adjust the content of the commentary or provide additional information to pique the user's interest. For example, if the user is not interested in a particular exhibit, the guide can introduce other related exhibits.

[1671] Feedback Improvements

[1672] server

[1673] The server generates feedback to improve the user experience based on the emotional data analyzed by the emotion engine and provides it to the museum, which can then use it to improve the exhibits and tour content.

[1674] In this way, the system of the present invention recognizes and reflects user emotions in real time, providing a more personalized and high-quality online museum experience.

[1675] The processing flow will be explained below.

[1676] 1. Digital archiving of museums

[1677] Step 1:

[1678] Server: Provides a dedicated form on the website for museum staff to upload digital data (images, videos, text, etc.) of exhibits.

[1679] Step 2:

[1680] Server: Receives uploaded digital data and moves it to a temporary storage directory. For example, if a museum uploads a high-resolution image of "Famous Painting A," the data is stored in that directory.

[1681] Step 3:

[1682] Server: Analyzes the received digital data and executes analysis scripts to extract metadata (artist name, year of production, technique, etc.). The extracted metadata is stored in a database along with the data.

[1683] Step 4:

[1684] Server: Indexes the data to enable quick searches and filtering later. The indexed data is efficiently managed in a database.

[1685] 2. Online tour guide matching

[1686] Step 1:

[1687] User: The user accesses the virtual museum website from their device, enters the tour guide request screen, and enters "I would like a tour of the Impressionists." This information is sent to the server.

[1688] Step 2:

[1689] Server: Analyzes the user's request, searches for guide information in the database, generates a list of target guides, and sends it to the user's device.

[1690] Step 3:

[1691] User: The user selects the desired guide from the displayed guides and sends the selected information to the server. For example, if the user selects "Guide B," the information is sent to the server.

[1692] Step 4:

[1693] Server: Requests confirmation and approval of the tour date and time from the selected guide, and sends the confirmation information to the user once approval is received.

[1694] 3. Conducting Virtual Tours

[1695] Step 1:

[1696] Server: When the tour start time arrives, the server sends a tour start notification to the user and guide's devices and preloads the digital data required for the tour.

[1697] Step 2:

[1698] Guide: The guide uses a dedicated application to operate the exhibits and provide commentary. Screen operations and commentary are distributed to users in real time via a server.

[1699] Step 3:

[1700] Users: Users can operate the virtual space on their own devices, view the exhibits in detail, and send questions to the guide using the chat function.

[1701] Step 4:

[1702] Server: Converts user questions into text and sends it to the guide. If necessary, translates the questions to support multiple languages.

[1703] Step 5:

[1704] Guide: The guide answers users' questions in real time.

[1705] 4. Secondary Features of the Service

[1706] Step 1:

[1707] Terminal: After the tour ends, the server displays a questionnaire on the user's terminal to collect feedback information. The user answers the questionnaire and sends the results to the server.

[1708] Step 2:

[1709] Server: Analyzes the collected survey results and comments made during the tour, generates a feedback report, and provides it to the museum.

[1710] Step 3:

[1711] Server: Generates information recommending products from the museum store based on the exhibits and artists that the user is interested in during the tour, and sends this information to the user's device.

[1712] Step 4:

[1713] Device: Display related products on the user's device and provide a purchase link. For example, related products for "Famous Painting A" are displayed, and the user can purchase them via a link to an e-commerce site.

[1714] 5. Incorporating an Emotional Engine

[1715] Step 1:

[1716] Device: Obtains facial expression data, tone of voice, text input, etc. in real time from the user's device.

[1717] Step 2:

[1718] Server: Sends the acquired data to the emotion engine to analyze the user's emotional state, for example, analyzing the emotions (excitement, indifference, etc.) when the user is viewing a particular exhibit.

[1719] Step 3:

[1720] Server: Based on the analysis results, the user's emotional state is fed back to the guide device in real time.

[1721] Step 4:

[1722] Guide: The guide dynamically adjusts its commentary based on the emotional feedback it receives, for example providing more detailed information about an exhibit that excites the user.

[1723] Step 5:

[1724] Server: After the tour ends, the emotion engine generates feedback based on the emotional data analyzed and provides it to the museum.

[1725] In this way, the specific processing content at each step ensures that users can have a high-quality experience through the virtual museum. In addition, by incorporating an emotion engine, a more personalized and engaging experience is provided.

[1726] Example 2

[1727] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1728] Conventional digital archive systems for museum exhibits simply store and display digital data of exhibits, limiting the user experience. They also lack digital tour guide recommendations and two-way communication functions, making it difficult to provide a personalized experience tailored to participants. Furthermore, they lack the ability to recognize users' emotions in real time and reflect them in the guide's commentary, making it difficult to respond flexibly to users' interests and level of understanding.

[1729] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for digitally archiving museum exhibits, means for displaying the digitized exhibits on a user terminal, means for recommending an optimal tour guide based on the user's interests and concerns, means for conducting an online tour in which the guide explains the exhibits, means for supporting two-way communication between the guide and the user in real time, means for analyzing the user's emotional data and providing feedback to the guide in real time, and means for collecting feedback from the user after the tour ends and providing the feedback to the museum. This allows the user to experience a high-quality virtual tour of the museum's exhibits through the digital archive, and receive a personalized explanation based on the real-time emotional data.

[1730] An "art museum" is a facility that displays exhibits for the general public and aims to promote education and culture.

[1731] "Exhibits" refers to cultural assets and artworks such as paintings, sculptures, crafts, and photographs displayed in art museums.

[1732] "Digital archiving" is the process of recording, preserving, and managing exhibits as digital data (images, videos, text, etc.).

[1733] "User Terminal" refers to a device, such as a computer, smartphone, or tablet, that a User uses to access the System via the Internet.

[1734] A "tour guide" is a professional or function that provides commentary and explanations about museum exhibits, either online or in person.

[1735] The "recommendation method" is part of a system that uses algorithms and filtering functions to select and suggest the most suitable tour guide based on the user's interests.

[1736] An "online tour" is a service that allows users to virtually view museum exhibits using the Internet while receiving real-time commentary from a guide.

[1737] "Two-way communication" refers to the interactive function that allows users and guides to exchange information in real time.

[1738] "Emotional data" refers to information about a user's psychological state and emotions analyzed from facial expressions, tone of voice, text input, etc.

[1739] "Feedback" refers to information such as opinions, impressions, and evaluations collected from users after the tour, and is used to improve and evaluate the service.

[1740] The present invention is a system that digitally archives museum exhibits and provides a virtual museum that users can access online. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system provides a more personalized experience. Specific embodiments of each function are described below.

[1741] Digital archiving of museum exhibits

[1742] server

[1743] The server receives digital data (images, videos, text, etc.) of exhibits provided by the museum and temporarily stores it. For example, if the museum uploads a high-resolution image of "Famous Painting A," the server receives this data and moves it to a temporary storage directory. The server analyzes the received digital data and extracts metadata (artist name, year of production, technique, etc.). Software such as image analysis algorithms is used for the analysis. The analyzed data is converted into an appropriate format and stored in a dedicated database. In addition, indexing the data allows for faster searching and filtering later.

[1744] Online tour guide matching

[1745] user

[1746] A user accesses the virtual museum from their device and requests a tour guide. For example, if they enter their request as "I would like to learn about Impressionism," this information is sent to the server. The server analyzes the user's request and searches for guide information in its database. For example, it creates a list of guides who are knowledgeable about "Impressionism" and sends this list to the user. The user selects the desired guide from the displayed guides and sends that information to the server. The server then requests confirmation and approval of the tour date and time from the selected guide, and if approval is granted, sends the confirmed information to the user.

[1747] Conducting a virtual tour

[1748] Server, user terminal, guide terminal

[1749] When the tour begins, the server sends a notification to the user's and guide's devices and pre-loads the digital data necessary for the tour. The guide uses a dedicated application to operate the exhibits while providing commentary. For example, the guide might display details of "Masterpiece A" on the screen while explaining its background and techniques. The guide's device sends the commentary and screen operation information to the user in real time via the server. Users can operate the virtual space on their own devices and view the exhibits in detail. Users can also send questions using the chat function. The server receives questions from users, converts them to text, and sends them to the guide. These questions are translated as needed, making multilingual support possible. The guide then answers the user's questions in real time.

[1750] Service Sub-Features

[1751] Server, user terminal

[1752] After the tour ends, the server displays a questionnaire on the user's device to collect feedback information. For example, feedback is collected by having the user answer the question, "How satisfied were you with this tour?" The collected feedback information is analyzed and provided to the museum. The server analyzes the collected feedback and the user's interests, generates information recommending related museum store products, and sends this information to the user's device. For example, related products for "Famous Painting A" are displayed, and the user can purchase them via a link to an e-commerce site.

[1753] Incorporating an emotion engine

[1754] Emotion engine, server

[1755] The emotion engine analyzes the user's facial expression data, tone of voice, and text input content obtained in real time from the user's device to recognize the user's emotional state. For example, it determines whether the user is excited or bored by a particular exhibit. This information is fed back to the guide device via the server. The guide then dynamically adjusts the content of the commentary based on this information. For example, if the user shows no interest in a particular exhibit, the guide will introduce other related exhibits. The server generates feedback to improve the user experience based on the emotional data analyzed by the emotion engine and provides it to the museum.

[1756] Examples of concrete examples and prompts

[1757] For example, if an art museum were to digitally archive an "Impressionist Exhibition," it would follow the steps below.

[1758] 1. The museum uploads a high-resolution image of "Masterpiece A" to the server.

[1759] 2. The server analyzes the image data and extracts metadata such as the artist's name and technique.

[1760] 3. The user requests, "I want to learn about Impressionism," and the server matches them with a suitable tour guide.

[1761] 4. When the tour start time arrives, the server notifies the user and guide devices and the virtual tour begins.

[1762] 5. The guide explains the details of the exhibits, and when the user asks questions, they are sent to the guide via the server.

[1763] 6. After the tour, a survey will be displayed on the user's device to collect feedback information.

[1764] Prompt Sentence Examples

[1765] I want to book a virtual tour about Impressionism.

[1766] I want to know more about Masterpiece A.

[1767] The above is an embodiment of the present invention, which allows users to experience a high-quality virtual tour and receive personalized commentary in real time.

[1768] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1769] Step 1:

[1770] The server receives the digital data from the museum.

[1771] Input: High-resolution images, videos, and text data uploaded by museums.

[1772] Data processing: The input data is saved in a temporary directory.

[1773] Output: Digital data saved in temporary directory.

[1774] Specific operation: When the museum uploads a high-resolution image of "Famous Painting A," the server receives this data and moves it to a temporary storage directory.

[1775] Step 2:

[1776] The server analyzes the received digital data and extracts the metadata.

[1777] Input: Digital data stored in the temporary directory.

[1778] Data processing: Analyze the digital data using software such as image analysis algorithms.

[1779] Output: Extracted metadata (artist name, year of production, technique, etc.).

[1780] Specific operation: The server analyzes the stored image data and automatically extracts information about the artist and era of the work.

[1781] Step 3:

[1782] The server converts the parsed data into an appropriate format and stores it in a database.

[1783] Input: Extracted metadata and original digital data.

[1784] Data processing: Image data is converted into a specific compressed format and stored in a database along with metadata.

[1785] Output: Digital data and metadata stored in a database.

[1786] Specific operation: The server converts the analyzed data into an appropriate format and stores the digital data and metadata in a database.

[1787] Step 4:

[1788] Users access the virtual museum from their own devices and request a tour guide.

[1789] Input: A tour guide request sent from the user's device (e.g., "I want to learn about Impressionism").

[1790] Data processing: The server receives and analyzes the user's request.

[1791] Output: The tour guide request sent to the server.

[1792] What it does: A user types in "I want to learn about Impressionism," and this information is sent to the server.

[1793] Step 5:

[1794] The server analyzes the tour guide request and matches the most suitable guide.

[1795] Input: User request data.

[1796] Data processing: The server searches the guide information in the database and lists related guides.

[1797] Output: A list of guides that will be sent to the user.

[1798] What it does: The server compiles a list of detailed guides to "Impressionism" from its database and sends this list to the user.

[1799] Step 6:

[1800] The user selects the desired guide and submits it to the server, which requests confirmation and approval of the tour date and time.

[1801] Input: User selected guide information.

[1802] Data processing: The server confirms the tour date and time with the selected guide and requests approval.

[1803] Output: Tour information approved by the guide.

[1804] Specific operation: The user selects the desired guide and sends the information to the server. The server requests the guide to confirm and approve the tour date and time, and once approval is received, the confirmation information is sent to the user.

[1805] Step 7:

[1806] At the start time of the tour, the server sends a notification and pre-loads the digital data required for the tour.

[1807] Input: Confirmed tour information.

[1808] Data processing: Preload the necessary digital data.

[1809] Output: Tour start notification to user and guide devices.

[1810] Specific operation: When the tour start time arrives, the server sends a notification and loads the digital data required for the tour into the user's and guide's devices in advance.

[1811] Step 8:

[1812] The guide will provide commentary while operating the exhibits, and users can send questions via their terminal.

[1813] Input: Data about the exhibits operated by the guide, user question data.

[1814] Data processing: The server sends the commentary and screen operation information to the user, and sends the user's questions to the guide.

[1815] Output: Instructions sent to the user, user questions sent to the guide.

[1816] Specific operation: The guide explains "Famous Painting A," and when the user asks a question, the server sends the question to the guide, who then answers in real time.

[1817] Step 9:

[1818] The emotion engine analyzes the user's emotional data and provides feedback to the guide via the server.

[1819] Input: User facial expression data, tone of voice, and text input.

[1820] Data processing: The emotion engine analyzes the emotion data and provides feedback to the guide via the server.

[1821] Output: The user's emotional state, which is fed back to the guide.

[1822] Specific operation: The emotion engine analyzes the user's emotions and provides the guide with the user's excitement level or boredom level in real time.

[1823] Step 10:

[1824] After the tour ends, the server displays a questionnaire on the user's device to collect feedback information.

[1825] Input: User survey data after the tour.

[1826] Data processing: Analyze the collected survey results and generate a feedback report to provide to the museum.

[1827] Output: Feedback report provided to the museum, related product recommendations displayed on the user's device.

[1828] Specific operation: After the tour ends, a questionnaire is displayed on the user's device, and feedback information is collected and analyzed. The analysis results are provided to the museum, and related product recommendations are displayed on the user's device.

[1829] (Application example 2)

[1830] Next, a description will be given of Application Example 2. In t...

Claims

1. A means of digitally archiving museum exhibits, a means for displaying the digitized exhibits on a user terminal; A means of recommending the most suitable tour guide based on the user's interests and preferences; A means of conducting online tours where guides will explain the exhibits; A system that includes a means of supporting two-way communication between the guide and the user in real time.

2. The system according to claim 1, further comprising means for converting the commentary content of the guide into text, translating the text, and providing the text to the user.

3. 10. The system of claim 1, further comprising means for collecting and providing feedback from users to the museum after the tour is completed.

Images