System

Abstract

A system is provided.SOLUTION: A system for enabling a remotely located user to view an exhibit of a cultural establishment in a guided manner, the system comprising: Means for generating a three dimensional virtual space based on a digital archive and appropriately arranging an exhibit in the virtual space, means for inputting and managing profile information and specialist field information of a guide, means for inputting and displaying an interest of a user and a desired exhibit content, means for matching an optimal guide and a user based on input guide information and user information, means for guiding the guide in the virtual space and providing an explanation in real time, means for collecting feedback from the user and providing the collected feedback to the guide and a cultural facility, and collecting a tour fee from the user, means for distributing the fee to the guide, the cultural establishment, and the system operator.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Many cultural facilities face challenges such as a lack of funding, delayed digitalization, and a shortage of personnel. Access to cultural facilities is also difficult for elderly people living far away and busy users. Furthermore, use of facilities is concentrated on holidays, and long lines form for popular exhibits, making it difficult to fully appreciate the exhibits. There is a need to solve these issues and increase revenue for cultural facilities and user satisfaction. [Means for solving the problem]

[0005] This system allows remote users to enjoy guided viewing of exhibits at cultural facilities. The system receives exhibit data provided by the cultural facility, stores it as a digital archive, generates a three-dimensional virtual space based on the digital archive, and appropriately arranges the exhibits within the virtual space. It also includes a means for inputting and managing guide profile information and specialty information, and a means for inputting and displaying user interests and desired exhibit content. It also includes a function for matching users with the most suitable guide based on the input guide and user information, and for the guide to guide the virtual space and provide commentary in real time. After the tour, the system collects and analyzes user feedback and provides the results to the guide and the cultural facility. It also includes a function for distributing tour fees collected from users to the guide, the cultural facility, and the system operator, thereby increasing the revenue of the cultural facility.

[0006] "Cultural facilities" refers to public or private exhibition spaces such as art galleries, museums, and galleries.

[0007] "Exhibit" refers to an item presented to visitors within a cultural institution, such as a work of art, historical artifact, or scientific model.

[0008] "User" refers to an individual who wishes to view exhibits at a cultural facility through the service.

[0009] A "guide" is an individual who has specialized knowledge about the exhibits at a cultural facility and provides explanations to visitors.

[0010] A "digital archive" refers to an electronic database that stores and manages digital data of exhibits provided by cultural facilities.

[0011] "3D virtual space" refers to a three-dimensional digital environment that imitates a real space and is generated using computer technology.

[0012] "Profile Information" refers to detailed information about a guide, such as their personal background, expertise, and achievements.

[0013] "Specialty Information" refers to information about an artistic field, historical period, or theme in which the guide has particular expertise.

[0014] "Interests" refers to the personal preferences of visitors regarding the exhibits or themes they wish to view.

[0015] "Matching" refers to the process of matching the most suitable guide with the user based on the user's interests and the guide's areas of expertise.

[0016] "Real-time" refers to information processing and communication that occurs instantly with almost no delay.

[0017] "Feedback" refers to opinions and ratings provided by users after the tour regarding the quality of the guide and their satisfaction with the tour.

[0018] "Collection" refers to the process of compiling and gathering information or data.

[0019] "Analysis" refers to the process of examining collected data in detail to extract meaningful information.

[0020] "Tour Fee" means the fee paid by a User for a guided tour.

[0021] "Distribution" refers to the process of distributing the collected tour fees to guides, cultural facilities, and system operators in predetermined proportions. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram illustrating a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0023] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0024] First, the terms used in the following description will be explained.

[0025] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0026] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0027] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0028] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0029] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0030] [First embodiment]

[0031] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0032] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0033] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0034] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0035] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0036] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0037] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0038] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0039] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0040] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0041] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0042] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0043] The system of the present invention allows users to view exhibits at cultural facilities remotely and provides guided tours. The system's program consists of numerous modules and tools, providing users with a high-quality cultural experience.

[0044] The main components of this system are as follows:

[0045] Digital archiving of art museums

[0046] server:

[0047] Images, videos, and text data of exhibits are received from cultural facilities via API. This data is stored on the server and analyzed to register metadata (title, artist, era, description, etc.) for each exhibit. As a specific example, the server receives image data of a "famous painting," extracts detailed information about it using image analysis technology, and stores it in a digital archive.

[0048] Creating a virtual museum

[0049] server:

[0050] The server generates a three-dimensional virtual space based on the stored digital archives. Within this virtual space, exhibits are arranged in the same way as they would be in a real cultural facility. For example, the server recreates the interior of a "historical building" and places the "ancient artifacts" on display in the appropriate locations.

[0051] Guide registration and management

[0052] Terminal (Guide):

[0053] Guides use a dedicated application to register their profile information (career history, expertise, achievements) and areas of expertise. This information is sent to the server and stored in a database. For example, a guide can enter a profile as a "historian," and that information is stored and managed within the system.

[0054] User registration and matching system

[0055] Terminal (user):

[0056] Users create an account through the application and enter their interests (the exhibits and themes they wish to view) and available time. The server matches them with the most suitable guide based on the information they enter. For example, if a user expresses an interest in "Renaissance paintings," the server will select a guide who is knowledgeable about Renaissance paintings.

[0057] Tour execution and feedback collection

[0058] Terminals (Guide and User):

[0059] At the specified date and time, the guide and user access the application. The guide guides the user through the virtual museum and provides commentary in real time. Users can share questions and impressions through chat or voice communication. After the tour ends, the user fills out a feedback form and sends it to the server. The server collects and analyzes the feedback and provides it to the guide and cultural facility. For example, while the guide explains to the user about "modern sculpture," the user is free to ask questions and provide feedback after the tour ends.

[0060] Revenue Share Distribution

[0061] server:

[0062] The server collects the tour fee from the user and distributes it to the guide, the cultural facility, and the system operator. For example, if the tour fee is paid by credit card, the server automatically distributes the fee to each party.

[0063] By integrating these functions, this system will solve the problems of cultural facilities, such as a lack of funding, delays in digitalization, and difficulty in accessing the system for users, and improve the quality of the cultural experience. Each function has been explained with concrete examples, and it is expected that this will increase the revenue of cultural facilities and improve user satisfaction.

[0064] The processing flow will be explained below.

[0065] Digital archiving of art museums

[0066] Step 1:

[0067] server:

[0068] Images, videos, and text data provided by cultural facilities are received via API.

[0069] Step 2:

[0070] server:

[0071] Store the received data in local storage or cloud storage.

[0072] Step 3:

[0073] server:

[0074] Metadata (title, artist, year, description) is extracted from the stored data using an image analysis algorithm.

[0075] Step 4:

[0076] server:

[0077] The extracted metadata is registered in a database.

[0078] Creating a virtual museum

[0079] Step 1:

[0080] server:

[0081] Using 3D modeling tools, a three-dimensional virtual space is generated based on the digital archive.

[0082] Step 2:

[0083] server:

[0084] The coordinate data of the exhibits is set within the generated virtual space.

[0085] Step 3:

[0086] server:

[0087] Based on the coordinate data, the exhibits are arranged and interactive functions are added.

[0088] Guide registration and management

[0089] Step 1:

[0090] Terminal (Guide):

[0091] The guide opens a dedicated application and enters their profile information and areas of expertise.

[0092] Step 2:

[0093] Terminal (Guide):

[0094] Also add detailed information such as your self-introduction and achievements.

[0095] Step 3:

[0096] server:

[0097] Receives guide input information and stores it in a database.

[0098] User registration and matching system

[0099] Step 1:

[0100] Terminal (user):

[0101] The user opens the dedicated application and creates an account.

[0102] Step 2:

[0103] Terminal (user):

[0104] Enter your interests, desired exhibits, and available time.

[0105] Step 3:

[0106] server:

[0107] It takes the user's input information and searches the database for the appropriate guide.

[0108] Step 4:

[0109] server:

[0110] Using an AI algorithm, it matches the user's interests, the guide's areas of expertise, and exhibit information.

[0111] Step 5:

[0112] server:

[0113] The matching results are presented to the user, and the user selects a guide.

[0114] Tour execution and feedback collection

[0115] Step 1:

[0116] Terminals (Guide and User):

[0117] The Guide and user access the application at the specified date and time.

[0118] Step 2:

[0119] Terminal (Guide):

[0120] A guide will guide you through the virtual museum and provide real-time commentary on the exhibits.

[0121] Step 3:

[0122] Terminal (user):

[0123] Users share questions and thoughts through chat functionality and voice communication.

[0124] Step 4:

[0125] Terminal (user):

[0126] After the tour, the user fills out a feedback form.

[0127] Step 5:

[0128] server:

[0129] Receive user feedback and store it in a database.

[0130] Step 6:

[0131] server:

[0132] The feedback data is analysed and provided to guides and cultural institutions as detailed reports.

[0133] Revenue Share Distribution

[0134] Step 1:

[0135] server:

[0136] The tour fee is collected from the user via credit card or electronic payment.

[0137] Step 2:

[0138] server:

[0139] Temporarily store the collected fees.

[0140] Step 3:

[0141] server:

[0142] The fees will be distributed to guide, cultural institution and platform accounts based on a revenue share percentage.

[0143] Step 4:

[0144] server:

[0145] Process payments to each distribution recipient.

[0146] Through these processing steps, the system of the present invention simultaneously realizes the digitization of cultural facilities, improved convenience for users, and increased profits.

[0147] Example 1

[0148] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0149] There is a need to provide the experience of viewing exhibits at cultural facilities to users in remote locations. Another challenge is to enable users to not only receive real-time commentary from guides, but also to select guides that suit their interests. Furthermore, it is necessary to collect and analyze user feedback and provide it to guides and cultural facilities to improve the quality of services. Finally, to ensure the economic sustainability of this system, an appropriate distribution of revenue shares must be addressed.

[0150] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0151] In this invention, the server includes: means for receiving exhibit data provided by the cultural facility and saving it as a digital archive; means for generating a three-dimensional virtual space based on the digital archive and appropriately arranging exhibits within the virtual space; means for inputting and managing guide profile information and specialty information; means for inputting and displaying user interests and desired exhibits; means for matching the optimal guide with the user based on the input guide information and user information; means for the guide and user to access the application at a specified date and time and provide real-time commentary; interactive means for users to share questions and impressions via chat or voice communication; means for collecting and analyzing feedback from users; means for providing the collected feedback to the guide and the cultural facility; and means for collecting tour fees from users and distributing the fees to the guide, the cultural facility, and the system operator. This allows users to view exhibits at the cultural facility in real time, even from remote locations, and enjoy an interactive experience tailored to their interests while listening to the guide's commentary. Furthermore, collecting and analyzing feedback and appropriately distributing fees improves the quality of the system and ensures its economic sustainability.

[0152] A "remotely located user" is a user who is physically located away from the cultural facility but who views the exhibits at the cultural facility through this system.

[0153] "Cultural facilities" refer to facilities that store and display exhibits, such as art galleries and museums.

[0154] "Exhibits" are objects or materials that are available for viewing, such as works of art, archaeological artifacts, or natural history exhibits, that are displayed at cultural institutions.

[0155] A "digital archive" is a database for digitally storing and managing data, including images, videos, and text of exhibits provided by cultural facilities.

[0156] A "three-dimensional virtual space" is a three-dimensional digital space generated by a computer, a virtual exhibition space that users can access and experience via the Internet.

[0157] A "guide" is a person with specialized knowledge who provides explanations and commentary on exhibits to visitors and leads tours.

[0158] "Profile information" refers to information about the guide's qualifications and abilities, such as the guide's background, expertise, and achievements.

[0159] "Specialty information" refers to information about areas or topics in which the guide is particularly knowledgeable.

[0160] "Interests" refer to the exhibits or themes that visitors are particularly interested in viewing.

[0161] "Desired exhibit content" refers to the specific exhibits or themes that the user would like to see on the tour.

[0162] "Matching" refers to the process of matching the most suitable guide with the user based on the guide information and user information entered.

[0163] "Application" refers to the software that allows guides and users to access the system and perform various operations.

[0164] "Interactive function" refers to a function that allows two-way communication, such as allowing users to ask questions about exhibits and obtain information in real time.

[0165] "Feedback" refers to impressions, evaluations, suggestions for improvement, and other information provided by users after the tour has ended.

[0166] "Feedback collection" refers to the process of collecting feedback provided by users as data.

[0167] "Feedback analysis" refers to the process of analyzing collected feedback to help improve the quality of services.

[0168] "Tour Fee" means the fee paid by a User to participate in a Guided Tour.

[0169] "Revenue share" is a system in which collected tour fees are distributed to guides, cultural facilities, and system operators.

[0170] MODE FOR CARRYING OUT THE INVENTION

[0171] The system of the present invention allows users in remote locations to view exhibits at cultural facilities and experience guided tours. This system operates in cooperation with a server, terminals, and users. A specific implementation method is described below.

[0172] Digital archiving of art museums

[0173] The server first receives data (images, videos, text, etc.) about exhibits from cultural facilities via an API. The received data is analyzed using image analysis software such as TensorFlow, and metadata such as the work's name, artist, era, and description is extracted. This data is then stored as a digital archive. For example, the server could receive image data of a "famous painting," extract its detailed information, and store it in a digital archive.

[0174] Creating a virtual museum

[0175] The server uses the stored digital archives to create a virtual space using 3D virtual space generation software such as Unity. Exhibits are placed in this virtual space just as they would be in a real cultural facility. For example, a historical building could be recreated, with ancient artifacts appropriately placed inside.

[0176] Guide registration and management

[0177] Guides use a dedicated terminal application to enter their profile information (career history, expertise, achievements) and areas of expertise. This information is sent to the server and stored in a database. For example, a guide enters their profile as a historian, and that information is stored in the system.

[0178] User registration and matching system

[0179] Users create an account through the terminal application and enter their interests, desired exhibits, and available times. Based on this information, the server selects the most suitable guide and matches them with the user. For example, if a user expresses an interest in Renaissance paintings, the server will select a guide who is an expert in that field.

[0180] Tour execution and feedback collection

[0181] At the specified date and time, the guide and user access the application. The guide guides the user through the virtual museum and provides real-time commentary using communication software such as Zoom or Microsoft Teams. Users can share questions and impressions through chat functions or voice communication. After the tour, the user fills out a feedback form and sends it to the server. The server collects and analyzes this feedback and provides it to the guide and cultural facility. For example, while the guide explains about modern sculpture to the user, the user can ask questions freely and provide their impressions after the tour.

[0182] Revenue Share Distribution

[0183] The server collects the tour fee from the user and distributes it to the guide, the cultural facility, and the system operator. For example, the server automatically distributes the tour fee paid by credit card to each party.

[0184] ---

[0185] Example prompt sentence:

[0186] 1. "I'm interested in Japanese history. I'd like to see exhibits from the Kamakura period. Which guide would be best?"

[0187] 2. "I'd like to tour a virtual art museum next Saturday. Please select a guide who specializes in 17th-century European painting."

[0188] 3. "I'd like a detailed explanation of the Mona Lisa. Could you please check the digital archive of this exhibit and explain it to me?"

[0189] This system allows users to view the rich exhibits at cultural facilities from remote locations and receive real-time commentary from expert guides. It also enables two-way communication, allowing for feedback to be used to improve the quality of the service. Furthermore, the system's sustainability is ensured through appropriate distribution of fees.

[0190] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0191] Program processing flow and steps

[0192] Digital archiving of art museums

[0193] Step 1:

[0194] The server receives images, videos, and text data of exhibits from cultural facilities via an API.

[0195] Input: Exhibit data from cultural facilities (images, videos, text)

[0196] Output: The received data is loaded into the server's memory.

[0197] Specific operation: The server calls the API and receives and displays exhibit data in real time.

[0198] Step 2:

[0199] The server analyzes the received data and extracts metadata using image analysis techniques such as TensorFlow.

[0200] Input: Received exhibit data

[0201] Output: Metadata such as title, artist, era, description, etc.

[0202] Specific operation: The server executes the image analysis model and displays the analysis progress on the monitoring screen.

[0203] Step 3:

[0204] The server stores the analyzed metadata in a digital archive.

[0205] Input: Extracted metadata

[0206] Output: Metadata stored in a digital archive database

[0207] Specific behavior: The server performs the save operation and logs a save completion message.

[0208] Creating a virtual museum

[0209] Step 1:

[0210] The server retrieves the necessary data from the stored digital archive.

[0211] Input: Digital Archive Database

[0212] Output: Loaded exhibit data

[0213] What happens: The server executes the database query and loads the retrieved data into memory.

[0214] Step 2:

[0215] The server generates a three-dimensional virtual space using Unity or similar software.

[0216] Input: Loaded exhibit data

[0217] Output: Generated 3D virtual space

[0218] Specific operation: The server starts the Unity engine and displays the progress of the virtual space generation in real time.

[0219] Step 3:

[0220] The server appropriately arranges the exhibits in the virtual space.

[0221] Input: Exhibit data, 3D virtual space

[0222] Output: The completed virtual museum

[0223] Specific operation: The server executes the data placement algorithm and logs a placement completion message.

[0224] Guide registration and management

[0225] Step 1:

[0226] The terminal (guide) uses a dedicated application to input profile information and area of ​​expertise information.

[0227] Input: Guide profile information, specialty information

[0228] Output: The information entered is displayed on the terminal.

[0229] What happens: The application displays a confirmation message and checks the validity of the input data.

[0230] Step 2:

[0231] The terminal (guide) sends the entered information to the server.

[0232] Input: Profile information, area of ​​expertise

[0233] Output: Guide information uploaded to the server

[0234] Specific behavior: The application displays the progress of data transmission and notifies you when transmission is complete.

[0235] Step 3:

[0236] The server stores the received guide information in a database.

[0237] Input: Guide profile information, specialty information

[0238] Output: Guide information stored in a database

[0239] Specific behavior: The server performs the save operation and logs a save completion message.

[0240] User registration and matching system

[0241] Step 1:

[0242] The terminal (user) creates an account through the application.

[0243] Input: User account information

[0244] Output: A confirmation message is displayed to create an account.

[0245] What happens: The application checks the validity of the information entered and displays a confirmation message.

[0246] Step 2:

[0247] The terminal (user) inputs their interests and desired exhibit contents.

[0248] Input: User's interests and desired exhibits

[0249] Output: The information entered is displayed on the terminal.

[0250] What happens: The application displays a confirmation message and checks the validity of the input data.

[0251] Step 3:

[0252] The terminal (user) sends input information to the server.

[0253] Input: User's interests and desired exhibits

[0254] Output: User information uploaded to the server

[0255] Specific behavior: The application displays the progress of data transmission and notifies you when transmission is complete.

[0256] Step 4:

[0257] The server selects the most suitable guide based on the input guide information and user information and matches them with the user.

[0258] Input: Guide information, user information

[0259] Output: The matched guide-user pair

[0260] What happens: The server runs the matching algorithm and logs a match completion message.

[0261] Tour execution and feedback collection

[0262] Step 1:

[0263] At the specified date and time, the terminals (guide and user) access the application.

[0264] Input: User and guide access information

[0265] Output: Successfully logged into the application

[0266] Specific behavior: The application performs login authentication and displays a login completion message.

[0267] Step 2:

[0268] The device (guide) will guide you through the virtual museum and provide real-time commentary using Zoom, Microsoft Teams, etc.

[0269] Input: Guide explanation, user access information

[0270] Output: Real-time commentary, video and audio data

[0271] Specific operation: The guide provides commentary and broadcasts video and audio data in real time.

[0272] Step 3:

[0273] Devices (users) share questions and impressions through chat functions and voice communication.

[0274] Input: Questions and comments from users

[0275] Output: Questions and comments for the guide

[0276] What it does: The application forwards your questions and comments to the guide in real time.

[0277] Step 4:

[0278] After the tour ends, the terminal (user) fills in a feedback form and sends it to the server.

[0279] Input: Feedback information

[0280] Output: Feedback sent to the server

[0281] What happens: The application displays the progress of the feedback submission and notifies you when it is complete.

[0282] Step 5:

[0283] The server collects and analyzes the feedback.

[0284] Input: Feedback data

[0285] Output: Parsed feedback information

[0286] What happens: The server runs the analysis algorithm and logs the results.

[0287] Revenue Share Distribution

[0288] Step 1:

[0289] The server collects the tour fee from the user.

[0290] Input: User's payment information

[0291] Output: Fees collected

[0292] Specific behavior: The server collects the fee through the payment gateway and logs a collection completion message.

[0293] Step 2:

[0294] The server distributes the collected fees to guides, cultural facilities, and system operators.

[0295] Input: Collected fee data

[0296] Output: Distributed fees to each party

[0297] Specific operation: The server executes the distribution algorithm and sends a distribution completion message to each participant.

[0298] (Application example 1)

[0299] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0300] Conventional exhibit viewing systems at cultural facilities have not fully realized remote guided viewing, which has led to issues such as users being unable to effectively enjoy the exhibits using the latest technology. Furthermore, two-way communication, appropriate feedback collection, and matching of optimal guides based on user interests are also insufficient, hindering progress in the digitalization of cultural facilities.

[0301] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0302] In this invention, the server includes means for receiving exhibit data provided by the cultural facility and saving it as a digital archive, means for generating a three-dimensional virtual space based on the digital archive and appropriately arranging exhibits within the virtual space, means for inputting and managing guide profile information and specialty information, means for inputting and displaying user interests and desired exhibit content, means for matching the optimal guide with the user based on the input guide information and user information, means for the guide to guide the virtual space and provide real-time commentary, two-way communication means for users to submit questions and comments about the exhibits in real time via smartphones or tablet devices, means for collecting and analyzing feedback from users, means for providing the collected feedback to the guide and the cultural facility, and means for collecting tour fees from users and distributing the fees to the guide, the cultural facility, and the system operator. This enables the provision of high-quality cultural experiences remotely, matching users with the optimal guide based on their interests, providing real-time commentary and two-way communication, and analyzing and providing feedback.

[0303] A "cultural facility" is a place that displays exhibits and provides education and entertainment to the public, including art galleries, museums, and galleries.

[0304] "Exhibits" are objects such as artworks, historical materials, documents, etc. that are displayed in cultural institutions and that convey cultural or academic knowledge.

[0305] A "digital archive" is a system in which images, videos, and text data of exhibits are stored in digital format and managed as a database for future reference and search.

[0306] A "three-dimensional virtual space" is a virtual three-dimensional environment generated using computer technology, in which users can move freely within the space and visually appreciate exhibits and buildings that do not physically exist.

[0307] "Profile information" refers to data that includes information such as a guide's background, expertise, and track record, and this information is used to select and match guides.

[0308] "Specialty information" is information about a field or theme in which the guide is particularly knowledgeable, which enables the guide to provide specialized commentary and guidance to users.

[0309] "Matching" refers to the process of matching the most suitable guide with the user based on the user's interests and desired exhibits, as well as the guide's profile information and area of ​​expertise.

[0310] "Real-time commentary" means that a guide provides real-time audio and text explanations about exhibits and themes while guiding users through a virtual space, allowing users to obtain information instantly.

[0311] "Two-way communication" is a function that allows users to send questions and comments to guides and systems in real time via smartphones or tablet devices, engaging in two-way communication.

[0312] "Feedback" refers to the opinions and impressions that users enter after completing a tour, and this data is used to improve guides and cultural facilities.

[0313] "Fee collection" refers to the process of receiving tour fees from users through their payment method and distributing those fees to guides, cultural facilities, and system operators through the system.

[0314] A "generative AI model" is an artificial intelligence model that learns from large amounts of data to generate language and analyze data in a natural, human-like manner, and is used to analyze user feedback and select the optimal guide.

[0315] A "prompt" is an instruction given to a generative AI model to perform a specific task, allowing the AI ​​model to analyze and generate based on the specified content.

[0316] This invention is a system that enables users in remote locations to view exhibits at cultural facilities with a guide. The program for this system comprises the following steps:

[0317] The server receives data (images, videos, text) about exhibits provided by cultural facilities and stores it as a digital archive. Specifically, the server uses image analysis technology (such as OpenCV) to extract detailed information and stores it in a database (such as SQLite). This makes it possible to manage metadata (such as the name of the work, artist, era, and description) for each exhibit.

[0318] Next, the server generates a 3D virtual space based on the digital archive. Exhibits are placed in this virtual space in the same way as they would be in a real cultural facility. For example, the server uses a 3D rendering engine such as Unity or Unreal Engine to recreate a virtual art gallery or museum. The server also manages the placement data of the exhibits and places them in the appropriate locations.

[0319] Next, the guide enters and registers their profile information (career history, expertise, achievements) and areas of expertise into the server using a dedicated application. This saves and manages the guide information in a database. For example, if a guide enters their profile as a "historian," this information is saved in the system.

[0320] Users can create an account through the application and enter their interests and desired exhibits. They can also enter the time they are available. This information is sent to the server and analyzed. The server then matches users with the most suitable guide based on their interests. For example, if a user expresses an interest in "Renaissance paintings," the server will select a guide with that expertise.

[0321] At the designated date and time, the guide and user access the application. The guide will guide the user through the virtual space and provide real-time commentary. Users can send questions and comments about the exhibits in real time via their smartphones or tablets. This two-way communication function allows users to view the exhibits while communicating directly with the guide.

[0322] After the tour ends, feedback is collected from users. The server analyzes the collected feedback and provides it to guides and cultural facilities. The generative AI model also analyzes the feedback and suggests improvements to guide behavior and optimal placement of exhibits. Furthermore, by inputting prompts such as "Generate optimal guide suggestions based on the user's interests and feedback" into the generative AI model, the selection of the optimal guide can be automated.

[0323] Regarding fee collection, tour fees are collected from users and distributed to guides, cultural facilities, and system operators. For example, fees can be collected using credit cards or electronic payment services (such as PayPal), and the system automatically distributes them to each party.

[0324] As a specific example, if User A uses the app and inputs that he or she is interested in Renaissance paintings, the server will match the user with Guide B, who has specialized knowledge, based on that information. The user accesses the app at the specified date and time, and Guide B will guide the user through the virtual museum. After the tour ends, User A provides feedback, which is saved on the server. The generative AI model then analyzes the feedback and suggests areas for improving Guide B's explanation and optimal placement of exhibits.

[0325] Example prompt: "Generate optimal guide suggestions based on the user's interests and feedback."

[0326] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0327] Step 1: Receive exhibit data from cultural facilities and store it as a digital archive

[0328] The server receives images, videos, and text data of exhibits provided by cultural facilities via an API. This data is sent to the server, where image analysis technology (e.g., OpenCV) is used to extract detailed information (such as the name of the work, the artist, the year, and a description) and store it in a database (e.g., SQLite). The input is data about the exhibits, and the output is digitally archived data.

[0329] Step 2: Generate a 3D virtual space based on the digital archive

[0330] The server generates a virtual space using a 3D rendering engine such as Unity or Unreal Engine based on the stored digital archive. Exhibits are placed in appropriate positions within the virtual space. The input is digital archive data, and the output is a 3D virtual space.

[0331] Step 3: Enter and manage your guide's profile information and specialty information

[0332] The guide uses a terminal to input his / her profile information (career history, expertise, achievements) and areas of expertise. This information is sent to the server and stored in a database. The input is the guide's profile information, and the output is the guide information stored in the database.

[0333] Step 4: Enter your interests and desired exhibits

[0334] Users create an account using their device and enter their interests (the exhibits and themes they wish to view) and the time they have available. This information is sent to the server and analyzed. The input is the user's interests and desired exhibits, and the output is the analyzed user information.

[0335] Step 5: Matching the best guide with the user

[0336] The server performs appropriate matching based on the input guide information and user information. Using a generative AI model, it inputs a prompt such as "Generate optimal guide suggestions based on the user's interests and feedback" and selects the optimal guide. The input is guide information and user information, and the output is the matched guide and user.

[0337] Step 6: A guide will guide you through the virtual space

[0338] At the specified date and time, the guide and user access the application using their terminals. The guide will guide the user through the virtual space in real time and provide explanations of the exhibits. The input is the guide and user matched with the specified date and time, and the output is the real-time guided tour.

[0339] Step 7: Send your questions and comments via interactive communication

[0340] Visitors can send questions and comments about the exhibits in real time via smartphones or tablet devices, and the guide will respond accordingly. The input is the visitor's questions and comments, and the output is the guide's responses.

[0341] Step 8: Collect and analyze feedback

[0342] After the tour ends, feedback is collected from the user. The server analyzes the collected feedback and provides it to the guide and cultural facilities. It also analyzes the feedback using a generative AI model and proposes improvements to the guide's behavior and optimal placement of exhibits. The input is the user's feedback, and the output is the analysis results and proposals.

[0343] Step 9: Toll collection and distribution

[0344] The server collects tour fees from users and distributes them to guides, cultural facilities, and system operators using credit card payments or electronic payment services (such as PayPal). The input is the user's payment information, and the output is the distributed fee.

[0345] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0346] The system of the present invention allows remote users to view exhibits at cultural facilities with a guide, and also combines an emotion engine that recognizes the user's emotions. The system's program consists of many modules and methods, which provide users with a high-quality cultural experience and promote the improvement of guides and cultural facilities.

[0347] The main components of this system are as follows:

[0348] Digital archiving of art museums

[0349] server:

[0350] Images, videos, and text data of exhibits are received from cultural facilities via API. This data is stored on a server, and metadata (such as the name of the work, artist, year, and description) for each exhibit is analyzed and saved as a digital archive. For example, image data of a "famous painting" is received, and detailed information is extracted using image analysis technology and saved in the digital archive.

[0351] Creating a virtual museum

[0352] server:

[0353] The server generates a three-dimensional virtual space based on the stored digital archives. Within this virtual space, exhibits are arranged in the same way as they would be in a real cultural facility. For example, the interior of a "historical building" can be recreated, with "ancient artifacts" on display placed in appropriate locations.

[0354] Guide registration and management

[0355] Terminal (Guide):

[0356] Guides use a dedicated application to register their profile information (career history, expertise, achievements) and areas of expertise. This information is sent to the server and stored in a database. For example, a guide can enter a profile as a "historian," and that information is stored and managed within the system.

[0357] User registration and matching system

[0358] Terminal (user):

[0359] Users create an account through the application and enter their interests (the exhibits and themes they wish to view) and available time. The server matches them with the most suitable guide based on the information they enter. For example, if a user expresses an interest in "Renaissance paintings," the server will select a guide who is knowledgeable about Renaissance paintings.

[0360] Tour execution and feedback collection

[0361] Terminals (Guide and User):

[0362] At the specified date and time, the guide and user access the application. The guide guides the user through the virtual museum and provides real-time commentary. Users can share questions and comments through chat or voice communication. For example, while the guide explains "modern sculpture" to the user, the user can freely ask questions and provide feedback after the explanation.

[0363] server:

[0364] The server receives feedback from users and stores it in a database. The server analyzes the feedback and provides detailed reports to guides and cultural facilities.

[0365] Incorporating an emotion engine

[0366] Terminal (user):

[0367] The system uses an emotion engine that analyzes the user's facial expressions, tone of voice, etc. to monitor the user's emotional state in real time. For example, if the user feels surprise or excitement after seeing an exhibit, that emotional state will be analyzed.

[0368] server:

[0369] The emotion engine analyzes the user's emotional data and provides feedback to the guide based on that information, such as instructing the guide to provide a more detailed explanation of an exhibit that the user is interested in.

[0370] Revenue Share Distribution

[0371] server:

[0372] The server collects the tour fee from the user and distributes it to the guide, the cultural facility, and the system operator. For example, if the tour fee is paid by credit card, the server automatically distributes the fee to each party.

[0373] By integrating these functions, this system simultaneously digitizes cultural facilities, improves user convenience, and increases revenue. Furthermore, by incorporating an emotion engine, it can enrich the user experience and provide data useful for improving the operation of guides and cultural facilities. This is expected to promote financial support and digitalization for cultural facilities, and improve user cultural satisfaction.

[0374] The processing flow will be explained below.

[0375] Digital archiving of art museums

[0376] Step 1:

[0377] server:

[0378] Images, videos, and text data provided by cultural facilities are received via API.

[0379] Step 2:

[0380] server:

[0381] Store the received data in local storage or cloud storage.

[0382] Step 3:

[0383] server:

[0384] Metadata (title, artist, year, description) is extracted from the stored data using an image analysis algorithm.

[0385] Step 4:

[0386] server:

[0387] The extracted metadata is registered in a database.

[0388] Creating a virtual museum

[0389] Step 1:

[0390] server:

[0391] Using 3D modeling tools, a three-dimensional virtual space is generated based on the digital archive.

[0392] Step 2:

[0393] server:

[0394] The coordinate data of the exhibits is set within the generated virtual space.

[0395] Step 3:

[0396] server:

[0397] Based on the coordinate data, the exhibits are arranged and interactive functions are added.

[0398] Guide registration and management

[0399] Step 1:

[0400] Terminal (Guide):

[0401] The guide opens a dedicated application and enters their profile information and areas of expertise.

[0402] Step 2:

[0403] Terminal (Guide):

[0404] Add details about yourself, your achievements, etc.

[0405] Step 3:

[0406] server:

[0407] Receives guide input information and stores it in a database.

[0408] User registration and matching system

[0409] Step 1:

[0410] Terminal (user):

[0411] The user opens the dedicated application and creates an account.

[0412] Step 2:

[0413] Terminal (user):

[0414] Enter your interests, desired exhibits, and available time.

[0415] Step 3:

[0416] server:

[0417] It takes the user's input information and searches the database for the appropriate guide.

[0418] Step 4:

[0419] server:

[0420] Using an AI algorithm, it matches the user's interests, the guide's areas of expertise, and exhibit information.

[0421] Step 5:

[0422] server:

[0423] The matching results are presented to the user, and the user selects a guide.

[0424] Tour execution and feedback collection

[0425] Step 1:

[0426] Terminals (Guide and User):

[0427] The Guide and user access the application at the specified date and time.

[0428] Step 2:

[0429] Terminal (Guide):

[0430] A guide will guide you through the virtual museum and provide real-time commentary on the exhibits.

[0431] Step 3:

[0432] Terminal (user):

[0433] Users share questions and thoughts through chat functionality and voice communication.

[0434] Step 4:

[0435] Terminal (user):

[0436] After the tour, the user fills out a feedback form.

[0437] Step 5:

[0438] server:

[0439] Receive user feedback and store it in a database.

[0440] Step 6:

[0441] server:

[0442] The feedback data is analysed and provided to guides and cultural institutions as detailed reports.

[0443] Incorporating an emotion engine

[0444] Step 1:

[0445] Terminal (user):

[0446] Activate an emotion engine that analyzes the user's facial expressions and tone of voice.

[0447] Step 2:

[0448] Terminal (user):

[0449] The emotion engine analyzes the user's emotional state in real time and sends the data to the server.

[0450] Step 3:

[0451] server:

[0452] It receives emotional data sent in real time and generates feedback to the guide based on the analysis results.

[0453] Step 4:

[0454] Terminal (Guide):

[0455] The guide receives real-time feedback and adjusts the content and progress of the explanation based on the user's emotional state.

[0456] Revenue Share Distribution

[0457] Step 1:

[0458] server:

[0459] The tour fee is collected from the user via credit card or electronic payment.

[0460] Step 2:

[0461] server:

[0462] Temporarily store the collected fees.

[0463] Step 3:

[0464] server:

[0465] The fees will be distributed to guide, cultural institution and platform accounts based on a revenue share percentage.

[0466] Step 4:

[0467] server:

[0468] Process payments to each distribution recipient.

[0469] Through these processing steps, the system of the present invention simultaneously digitizes cultural facilities, improves user convenience, and increases revenue. Furthermore, by incorporating an emotion engine, it can enrich the user experience and provide data that is useful for improving the operation of guides and cultural facilities.

[0470] Example 2

[0471] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0472] When remote visitors view exhibits at cultural facilities, they need a high-quality, immersive experience while also being able to effectively collect, manage, and utilize their feedback on guides and cultural facilities. A system is also needed that can grasp the emotional state of visitors in real time and enable guides to respond appropriately based on that information. Fair and efficient distribution of revenue is also required.

[0473] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0474] In this invention, the server includes means for receiving exhibit data provided by the cultural facility and saving it as a digital archive, means for generating a three-dimensional virtual space based on the digital archive and appropriately arranging the exhibits within the virtual space, and means for inputting and managing guide profile information and specialty information. This allows remote users to enjoy a high-quality, immersive cultural experience while also enabling monitoring of the users' emotional states and real-time responses, and realizing fair and efficient distribution of revenue.

[0475] "Exhibits" refers to artworks, historical artifacts, artifacts, and other cultural assets that are made available to the public at cultural institutions.

[0476] A "guide" refers to a person who has knowledge of the exhibits at a cultural facility and provides explanations to visitors.

[0477] "User" refers to an individual in a remote location who wishes to view the exhibits at a cultural facility.

[0478] A "digital archive" refers to a system that stores and manages exhibit data (images, videos, text, etc.) in digital format.

[0479] "Three-dimensional virtual space" refers to a three-dimensional virtual space generated using digital technology that recreates the interior of an actual cultural facility.

[0480] "Profile information" refers to information about the guide, such as their background, expertise, and achievements.

[0481] "Emotional state" refers to emotions (surprise, excitement, joy, etc.) analyzed from the user's facial expressions and tone of voice.

[0482] "Feedback" refers to impressions, opinions, and ratings collected from users regarding exhibits and guides.

[0483] "Distribution" refers to the fair and efficient division of revenues (e.g., tour fees) among guides, cultural institutions, and system operators.

[0484] The system of the present invention allows remote users to view exhibits at cultural facilities with a guide, and also combines an emotion engine that recognizes the user's emotions. The system is composed of multiple modules and means, providing users with a high-quality cultural experience and promoting the improvement of guides and cultural facilities.

[0485] Digital archiving of art museums

[0486] server:

[0487] Images, videos, and text data of exhibits are received from cultural facilities via API. For example, TensorFlow is used as a specific image analysis technology to extract the name of the work and information about the artist from the received image data. This data is stored on a server as metadata (work name, artist, year, description, etc.) and managed as a digital archive. As a specific example, image data of a famous painting is received, and the name of the work and information about the artist are extracted using image analysis technology.

[0488] Creating a virtual museum

[0489] server:

[0490] A 3D virtual space is generated based on the stored digital archives. 3D models of the exhibits are created using 3D engines such as Unity or Unreal Engine, and then placed inside the virtual museum. Specifically, the interior of a "historical building" is recreated, and the "ancient artifacts" to be displayed there are placed in the appropriate locations.

[0491] Guide registration and management

[0492] Terminal (navigator):

[0493] Guides use a dedicated application to register their profile information (career history, expertise, achievements) and areas of expertise. This information is sent to the server and stored in a database. For example, if a guide enters their profile as a "historian," the information is sent to the server in real time and stored in the database.

[0494] User registration and matching system

[0495] Terminal (user):

[0496] Users create an account through the application and input their interests (the exhibits and themes they wish to view) and available time. The server matches them with the most suitable guide based on the information they input. For example, if a user expresses an interest in "Renaissance paintings," the server will select a guide who is knowledgeable about Renaissance paintings.

[0497] Tour execution and feedback collection

[0498] Terminals (instructor and user):

[0499] At the designated date and time, the guide and user access the application. The guide will guide the user through the virtual museum and provide real-time commentary. Users can share their questions and impressions through chat and voice communication. For example, while the guide is explaining about "modern sculpture," users can freely ask questions and provide feedback after the explanation.

[0500] server:

[0501] Feedback from users is received and stored in a database. The server analyzes the feedback and provides detailed reports to guides and cultural facilities.

[0502] Incorporating an emotion engine

[0503] Terminal (user):

[0504] The system uses an emotion engine that analyzes the user's facial expressions and tone of voice to monitor the user's emotional state in real time. For example, the system uses the emotion recognition API from Microsoft Azure Cognitive Services as a specific analysis technology. For example, if a user feels surprise or excitement when looking at an exhibit, that emotional state is analyzed.

[0505] server:

[0506] The emotion engine analyzes the user's emotional data and provides feedback to the guide based on that information. For example, it can instruct the guide to provide a more detailed explanation of an exhibit that the user is interested in.

[0507] Revenue Share Distribution

[0508] server:

[0509] The server collects the tour fee from the user and distributes it to the guide, the cultural facility, and the system operator. Specifically, it uses a payment gateway API such as Stripe for payment processing. When the tour fee is paid by credit card, the server automatically distributes the fee to each party.

[0510] Specific examples and examples of prompts for generative AI models

[0511] As a concrete example, consider a case where a user wants to view a "famous Renaissance painting." In this case, the server selects a "guide with expertise in Renaissance painting." If the emotion engine detects the user's excitement, it instructs the guide to provide more detailed information about the exhibit. An example of a prompt sentence for the generative AI model is as follows:

[0512] 1. "User is interested in Renaissance paintings. Please select relevant guides."

[0513] 2. "The user seems surprised by the exhibit. Please instruct the guide to provide a detailed explanation."

[0514] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0515] Step 1:

[0516] The server receives and stores data from the cultural institution.

[0517] Input: Images, videos, and text data of exhibits provided by cultural facilities via API

[0518] Specific operation: The server receives data on the exhibits using an API. To do this, it uses image analysis technology such as TensorFlow to extract the name of the work and information about the artist from the received image data and saves it as metadata.

[0519] Output: The analyzed metadata and digital data of the exhibits are saved on the server.

[0520] Step 2:

[0521] The server generates a three-dimensional virtual space.

[0522] Input: Stored digital archive data

[0523] How it works: The server uses a 3D engine such as Unity or Unreal Engine to generate a 3D virtual space based on the stored digital archive data. 3D models of the exhibits are generated and placed appropriately within the virtual museum.

[0524] Output: A 3D virtual museum space accessible to visitors and guides

[0525] Step 3:

[0526] The device (guide) registers and manages profile information

[0527] Input: Guide's profile information (career history, expertise, achievements) and area of ​​expertise

[0528] Specific operation: The guide uses a dedicated application to enter his / her profile information and send it to the server. This information is saved in real time on the server and managed in a database.

[0529] Output: Guide profile information stored on the server

[0530] Step 4:

[0531] The device (user) inputs interest information

[0532] Input: User's interests (exhibits and themes they wish to see) and available time input through the application

[0533] How it works: A user uses the application to input the exhibits and themes they wish to see, as well as the time they have available. This information is sent to the server and stored in a database.

[0534] Output: User interest information and available time stored on the server

[0535] Step 5:

[0536] The server matches guides with users.

[0537] Input: Guide profile information and user interest information stored in the database

[0538] How it works: The server collates the user's interests and automatically selects the most suitable guide. This process may involve the use of generative AI models. For example, a prompt might read, "The user is interested in Renaissance paintings. Please select a relevant guide."

[0539] Output: A properly matched guide-user pair

[0540] Step 6:

[0541] Conducting guided tours and providing real-time commentary

[0542] Input: Pre-matched guide and user, 3D virtual space data

[0543] Specific operation: At the specified date and time, the guide and user access the application and begin a tour of the virtual museum. The guide explains the exhibits in real time, and users share their questions and impressions using the chat function and voice communication.

[0544] Output: Real-time commentary and user questions and comments

[0545] Step 7:

[0546] The server collects and analyzes the feedback

[0547] Input: User feedback (chat messages, impressions, ratings)

[0548] Specific operation: After the tour ends, the server collects feedback from users and stores it in a database. It then analyzes the feedback and provides a detailed report to the guide and the cultural facility.

[0549] Output: Parsed feedback report

[0550] Step 8:

[0551] Monitor emotional states using the Emotion Engine

[0552] Input: Real-time user facial expressions and tone of voice

[0553] Specific behavior: Using an emotion engine (e.g., Microsoft Azure Cognitive Services), the user's facial expressions and tone of voice are analyzed in real time to monitor their emotional state. The analysis results are sent to the server. An example prompt might be, "The user seems surprised by the exhibits. Please instruct the guide to provide a detailed explanation."

[0554] Output: Real-time analyzed emotion data

[0555] Step 9:

[0556] The server provides feedback based on emotion data

[0557] Input: Parsed emotion data

[0558] Specific operation: The server provides real-time feedback to the guide based on the analyzed emotion data, allowing the guide to provide more detailed information about the exhibits that the user is interested in.

[0559] Output: Feedback instructions based on emotion data

[0560] Step 10:

[0561] The server collects and distributes the tour fee.

[0562] Input: Tour payment information from user

[0563] Specific operation: The server uses a payment gateway API such as Stripe to collect tour fees from users, and distributes the collected fees fairly and efficiently among guides, cultural facilities, and system operators.

[0564] Output: Tour fare distributed to each party

[0565] (Application example 2)

[0566] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0567] Conventional systems for remote viewing of exhibits at cultural facilities have had the problem of being unable to provide a personalized experience based on the user's emotions and interests. In addition, they lacked the functionality for users to obtain information interactively about the exhibits, which could result in a decrease in the user's understanding of the explanations and a decrease in their satisfaction.

[0568] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for receiving exhibit data provided by a cultural facility and saving it as a digital archive; means for generating a three-dimensional virtual space based on the digital archive and appropriately arranging exhibits in the virtual space; means for inputting and managing guide profile information and specialty information; means for inputting and displaying the user's interests and desired exhibit content; means for matching the optimal guide with the user based on the input guide information and user information; means for the guide to guide the virtual space and provide commentary in real time; means for collecting and analyzing feedback from the user; means for providing the collected feedback to the guide and the cultural facility; means for collecting tour fees from the user and distributing the fees to the guide, the cultural facility, and the system operator; and means for analyzing the user's emotions and adjusting the guide's commentary based on the analysis data. This allows users to obtain information about the exhibits interactively and provides a personalized experience tailored to the user's emotions and interests.

[0569] "Cultural facilities" are facilities that collect, preserve, and display cultural exhibits and materials, such as art galleries, museums, and libraries.

[0570] An "exhibit" is a tangible item, such as a work of art, historical artifact, scientific specimen, or document, that is displayed to visitors at a cultural institution.

[0571] A "digital archive" is an electronic database for storing and managing images, videos, text data, etc. of exhibits in digital format.

[0572] A "three-dimensional virtual space" is a three-dimensional virtual space generated by a computer, and is a virtual environment that a user can experience using a virtual reality device or the like.

[0573] A "guide" is a person or system that has specialized knowledge about cultural facilities and exhibits and provides explanations and guidance to users.

[0574] "Profile Information" refers to personal information about a guide, such as their background, expertise, achievements, and areas of expertise.

[0575] "Interest" refers to the degree of interest a visitor has in exhibits or themes that they find particularly appealing.

[0576] "Matching" is the process of selecting the most suitable guide based on the user's interests and desired exhibits, and connecting the user with the guide.

[0577] "Feedback" refers to reactions such as impressions, evaluations, and opinions from users, and is the process of collecting and analyzing these to help improve services.

[0578] "Emotion analysis" is a technology that analyzes a user's facial expressions and tone of voice to assess their emotional state in real time.

[0579] "Personalization" means individually tailoring and providing services and content to suit each user's interests and emotions.

[0580] An embodiment of the present invention provides a remote guided viewing experience of exhibits at a cultural facility by linking a server, terminals (guides, users), and an emotion analysis system. Specific embodiments are described below.

[0581] 1. Digital Archiving

[0582] The server receives images, videos, and text data of exhibits provided by cultural facilities via an API. This data is stored in a database on the server, and metadata (such as the title of the work, artist, year, and description) for each exhibit is analyzed and saved as a digital archive. Specifically, a cloud server on AWS (Amazon Web Services) is used, and the saved data is managed in an SQL database.

[0583] 2. Creating a virtual store

[0584] The server generates a 3D virtual space based on the stored digital archives. Unity and Unreal Engine are used to appropriately position exhibits within this virtual space. This virtually recreates the layout of the actual cultural facility, providing users with a realistic experience.

[0585] 3. Guide registration and management

[0586] Guides use a dedicated application to enter their profile information (career history, expertise, achievements) and areas of expertise, and then send this information to a server. This data is stored in a database on the server and later used to match users. The device can be a smartphone, tablet, or PC.

[0587] 4. User Registration and Matching System

[0588] Users create an account through the application and enter their interests (exhibits and themes they wish to see) and available times. The server analyzes this information and matches them with the most suitable guide. In particular, frameworks such as React and AngularJS are used for the user interface.

[0589] 5. Product Tour and Feedback Collection

[0590] At a specified date and time, the guide and user access the application. The guide navigates the virtual space and provides commentary in real time. Users share questions and impressions through chat and voice communication. Commentary is provided using a combination of speech recognition and text conversion systems, and feedback is sent to a server for storage. Python and TensorFlow are used for feedback analysis.

[0591] 6. Incorporating an Emotion Analysis System

[0592] An emotion engine that analyzes the user's facial expressions and tone of voice is installed on the device to monitor the user's emotional state in real time. This is done using OpenCV and Azure Emotion API. Emotion data is sent to a server, which provides feedback to the guide based on the analysis results.

[0593] 7. Revenue Share Distribution

[0594] The server collects the tour fee from the user and automatically distributes it to the guide, the cultural facility, and the system operator. Payment is processed using electronic payment services such as Stripe and PayPal.

[0595] Specific examples

[0596] For example, if a user expresses interest in a "historical building" exhibit in a virtual store, the system will use that information to match with a guide knowledgeable in history and conduct a tour. If the user shows excitement at an "ancient relic" displayed during the tour, the emotion engine will analyze this emotion and prompt the guide to provide a detailed explanation.

[0597] Example prompts for generative AI models

[0598] "Analyze how users are responding to this product. Based on image and audio data, determine whether their reactions are positive or negative based on facial expressions and tone of voice, and reflect this in a detailed guide."

[0599] This allows visitors to interact with the exhibits and obtain information about them, and also enables a personalized experience based on the visitors' emotions and interests.

[0600] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0601] Step 1:

[0602] The server receives images, videos, and text data of exhibits provided by cultural facilities via API. The input data is the exhibit's metadata (work title, artist, year, description, etc.), which is stored in a database. The server analyzes the stored data and organizes it into a digital archive. Specifically, it performs an INSERT operation on the database table, saves the data, and generates a CONFIRM message.

[0603] Step 2:

[0604] The server generates a three-dimensional virtual space using Unity or Unreal Engine based on the saved digital archive. The input data is the digital archive data saved in step 1. This also includes location and layout information for the exhibits to be placed appropriately within the virtual space. Specifically, the server uses Unity's Scene Builder function to place the 3D models of the exhibits within the scene and generate EXPORTED scene data.

[0605] Step 3:

[0606] The terminal (guide) uses a dedicated application to input their own profile information (career history, expertise, achievements) and areas of expertise. The input data is the guide's personal information and expertise information, and is sent to the server. The server stores the sent data in a database. Specifically, the terminal receives the input data through the GUI, sends a POST request to the server, and the server inserts it into the database.

[0607] Step 4:

[0608] The device (user) creates an account through the application and inputs their interests (exhibits and themes they wish to view) and available times. The input data is the user's personal information and interest data, and is sent to the server. The server analyzes this and matches them with the most suitable guide. Specifically, the server applies guide information and a matching algorithm based on the user's interest data to generate MATCHED parameters.

[0609] Step 5:

[0610] At the specified date and time, the guide and user access the application. The guide navigates the virtual space and provides commentary in real time. The input data is the scene data of the virtual space and the commentary, which is displayed and played on the device. Specifically, the device renders the virtual space data and displays the guide's commentary in audio or text.

[0611] Step 6:

[0612] An emotion engine that analyzes the user's facial expressions and tone of voice is installed on the device, monitoring the user's emotional state in real time. The input data is images of the user's facial expressions and voice data, which is analyzed to generate emotional state data. Specifically, the device acquires data from the camera and microphone, runs an emotion analysis algorithm (OpenCV or Azure Emotion API), and sends the analysis results to the server.

[0613] Step 7:

[0614] The server provides feedback to the guide based on the emotion analysis results sent by the user. The input data is emotional state data, and based on this, feedback data is generated to adjust the guide's explanations. Specifically, the server analyzes the emotion data, generates a feedback message instructing the necessary action (strengthening the explanation, providing more detailed explanations, etc.), and sends a push notification to the guide's device.

[0615] Step 8:

[0616] After the tour ends, the server collects feedback from users and stores it in a database. The input data is the user's feedback information, which is analyzed to generate report data to be provided to cultural facilities and guides. Specifically, the server receives the feedback information, applies an analytical algorithm (such as natural language processing) to create a report, and sends it to the relevant parties.

[0617] Step 9:

[0618] The server collects tour fees from users and automatically distributes them to guides, cultural facilities, and system operators. The input data is payment information and distribution ratio data, and distribution transactions are carried out based on this. Specifically, the server collects fees through a payment gateway (Stripe or PayPal) and automatically makes payments to the parties involved based on the pre-set distribution ratio.

[0619] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0620] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0621] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0622] [Second embodiment]

[0623] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0624] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0625] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0626] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0627] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0628] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0629] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0630] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0631] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0632] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0633] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0634] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0635] The system of the present invention allows users to view exhibits at cultural facilities remotely and provides guided tours. The system's program consists of numerous modules and tools, providing users with a high-quality cultural experience.

[0636] The main components of this system are as follows:

[0637] Digital archiving of art museums

[0638] server:

[0639] Images, videos, and text data of exhibits are received from cultural facilities via API. This data is stored on the server and analyzed to register metadata (title, artist, era, description, etc.) for each exhibit. As a specific example, the server receives image data of a "famous painting," extracts detailed information about it using image analysis technology, and stores it in a digital archive.

[0640] Creating a virtual museum

[0641] server:

[0642] The server generates a three-dimensional virtual space based on the stored digital archives. Within this virtual space, exhibits are arranged in the same way as they would be in a real cultural facility. For example, the server recreates the interior of a "historical building" and places the "ancient artifacts" on display in the appropriate locations.

[0643] Guide registration and management

[0644] Terminal (Guide):

[0645] Guides use a dedicated application to register their profile information (career history, expertise, achievements) and areas of expertise. This information is sent to the server and stored in a database. For example, a guide can enter a profile as a "historian," and that information is stored and managed within the system.

[0646] User registration and matching system

[0647] Terminal (user):

[0648] Users create an account through the application and enter their interests (the exhibits and themes they wish to view) and available time. The server matches them with the most suitable guide based on the information they enter. For example, if a user expresses an interest in "Renaissance paintings," the server will select a guide who is knowledgeable about Renaissance paintings.

[0649] Tour execution and feedback collection

[0650] Terminals (Guide and User):

[0651] At the specified date and time, the guide and user access the application. The guide guides the user through the virtual museum and provides commentary in real time. Users can share questions and impressions through chat or voice communication. After the tour ends, the user fills out a feedback form and sends it to the server. The server collects and analyzes the feedback and provides it to the guide and cultural facility. For example, while the guide explains to the user about "modern sculpture," the user is free to ask questions and provide feedback after the tour ends.

[0652] Revenue Share Distribution

[0653] server:

[0654] The server collects the tour fee from the user and distributes it to the guide, the cultural facility, and the system operator. For example, if the tour fee is paid by credit card, the server automatically distributes the fee to each party.

[0655] By integrating these functions, this system will solve the problems of cultural facilities, such as a lack of funding, delays in digitalization, and difficulty in accessing the system for users, and improve the quality of the cultural experience. Each function has been explained with concrete examples, and it is expected that this will increase the revenue of cultural facilities and improve user satisfaction.

[0656] The processing flow will be explained below.

[0657] Digital archiving of art museums

[0658] Step 1:

[0659] server:

[0660] Images, videos, and text data provided by cultural facilities are received via API.

[0661] Step 2:

[0662] server:

[0663] Store the received data in local storage or cloud storage.

[0664] Step 3:

[0665] server:

[0666] Metadata (title, artist, year, description) is extracted from the stored data using an image analysis algorithm.

[0667] Step 4:

[0668] server:

[0669] The extracted metadata is registered in a database.

[0670] Creating a virtual museum

[0671] Step 1:

[0672] server:

[0673] Using 3D modeling tools, a three-dimensional virtual space is generated based on the digital archive.

[0674] Step 2:

[0675] server:

[0676] The coordinate data of the exhibits is set within the generated virtual space.

[0677] Step 3:

[0678] server:

[0679] Based on the coordinate data, the exhibits are arranged and interactive functions are added.

[0680] Guide registration and management

[0681] Step 1:

[0682] Terminal (Guide):

[0683] The guide opens a dedicated application and enters their profile information and areas of expertise.

[0684] Step 2:

[0685] Terminal (Guide):

[0686] Also add detailed information such as your self-introduction and achievements.

[0687] Step 3:

[0688] server:

[0689] Receives guide input information and stores it in a database.

[0690] User registration and matching system

[0691] Step 1:

[0692] Terminal (user):

[0693] The user opens the dedicated application and creates an account.

[0694] Step 2:

[0695] Terminal (user):

[0696] Enter your interests, desired exhibits, and available time.

[0697] Step 3:

[0698] server:

[0699] It takes the user's input information and searches the database for the appropriate guide.

[0700] Step 4:

[0701] server:

[0702] Using an AI algorithm, it matches the user's interests, the guide's areas of expertise, and exhibit information.

[0703] Step 5:

[0704] server:

[0705] The matching results are presented to the user, and the user selects a guide.

[0706] Tour execution and feedback collection

[0707] Step 1:

[0708] Terminals (Guide and User):

[0709] The Guide and user access the application at the specified date and time.

[0710] Step 2:

[0711] Terminal (Guide):

[0712] A guide will guide you through the virtual museum and provide real-time commentary on the exhibits.

[0713] Step 3:

[0714] Terminal (user):

[0715] Users share questions and thoughts through chat functionality and voice communication.

[0716] Step 4:

[0717] Terminal (user):

[0718] After the tour, the user fills out a feedback form.

[0719] Step 5:

[0720] server:

[0721] Receive user feedback and store it in a database.

[0722] Step 6:

[0723] server:

[0724] The feedback data is analysed and provided to guides and cultural institutions as detailed reports.

[0725] Revenue Share Distribution

[0726] Step 1:

[0727] server:

[0728] The tour fee is collected from the user via credit card or electronic payment.

[0729] Step 2:

[0730] server:

[0731] Temporarily store the collected fees.

[0732] Step 3:

[0733] server:

[0734] The fees will be distributed to guide, cultural institution and platform accounts based on a revenue share percentage.

[0735] Step 4:

[0736] server:

[0737] Process payments to each distribution recipient.

[0738] Through these processing steps, the system of the present invention simultaneously realizes the digitization of cultural facilities, improved convenience for users, and increased profits.

[0739] Example 1

[0740] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0741] There is a need to provide the experience of viewing exhibits at cultural facilities to users in remote locations. Another challenge is to enable users to not only receive real-time commentary from guides, but also to select guides that suit their interests. Furthermore, it is necessary to collect and analyze user feedback and provide it to guides and cultural facilities to improve the quality of services. Finally, to ensure the economic sustainability of this system, an appropriate distribution of revenue shares must be addressed.

[0742] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0743] In this invention, the server includes: means for receiving exhibit data provided by the cultural facility and saving it as a digital archive; means for generating a three-dimensional virtual space based on the digital archive and appropriately arranging exhibits within the virtual space; means for inputting and managing guide profile information and specialty information; means for inputting and displaying user interests and desired exhibits; means for matching the optimal guide with the user based on the input guide information and user information; means for the guide and user to access the application at a specified date and time and provide real-time commentary; interactive means for users to share questions and impressions via chat or voice communication; means for collecting and analyzing feedback from users; means for providing the collected feedback to the guide and the cultural facility; and means for collecting tour fees from users and distributing the fees to the guide, the cultural facility, and the system operator. This allows users to view exhibits at the cultural facility in real time, even from remote locations, and enjoy an interactive experience tailored to their interests while listening to the guide's commentary. Furthermore, collecting and analyzing feedback and appropriately distributing fees improves the quality of the system and ensures its economic sustainability.

[0744] A "remotely located user" is a user who is physically located away from the cultural facility but who views the exhibits at the cultural facility through this system.

[0745] "Cultural facilities" refer to facilities that store and display exhibits, such as art galleries and museums.

[0746] "Exhibits" are objects or materials that are available for viewing, such as works of art, archaeological artifacts, or natural history exhibits, that are displayed at cultural institutions.

[0747] A "digital archive" is a database for digitally storing and managing data, including images, videos, and text of exhibits provided by cultural facilities.

[0748] A "three-dimensional virtual space" is a three-dimensional digital space generated by a computer, a virtual exhibition space that users can access and experience via the Internet.

[0749] A "guide" is a person with specialized knowledge who provides explanations and commentary on exhibits to visitors and leads tours.

[0750] "Profile information" refers to information about the guide's qualifications and abilities, such as the guide's background, expertise, and achievements.

[0751] "Specialty information" refers to information about areas or topics in which the guide is particularly knowledgeable.

[0752] "Interests" refer to the exhibits or themes that visitors are particularly interested in viewing.

[0753] "Desired exhibit content" refers to the specific exhibits or themes that the user would like to see on the tour.

[0754] "Matching" refers to the process of matching the most suitable guide with the user based on the guide information and user information entered.

[0755] "Application" refers to the software that allows guides and users to access the system and perform various operations.

[0756] "Interactive function" refers to a function that allows two-way communication, such as allowing users to ask questions about exhibits and obtain information in real time.

[0757] "Feedback" refers to impressions, evaluations, suggestions for improvement, and other information provided by users after the tour has ended.

[0758] "Feedback collection" refers to the process of collecting feedback provided by users as data.

[0759] "Feedback analysis" refers to the process of analyzing collected feedback to help improve the quality of services.

[0760] "Tour Fee" means the fee paid by a User to participate in a Guided Tour.

[0761] "Revenue share" is a system in which collected tour fees are distributed to guides, cultural facilities, and system operators.

[0762] MODE FOR CARRYING OUT THE INVENTION

[0763] The system of the present invention allows users in remote locations to view exhibits at cultural facilities and experience guided tours. This system operates in cooperation with a server, terminals, and users. A specific implementation method is described below.

[0764] Digital archiving of art museums

[0765] The server first receives data (images, videos, text, etc.) about exhibits from cultural facilities via an API. The received data is analyzed using image analysis software such as TensorFlow, and metadata such as the work's name, artist, era, and description is extracted. This data is then stored as a digital archive. For example, the server could receive image data of a "famous painting," extract its detailed information, and store it in a digital archive.

[0766] Creating a virtual museum

[0767] The server uses the stored digital archives to create a virtual space using 3D virtual space generation software such as Unity. Exhibits are placed in this virtual space just as they would be in a real cultural facility. For example, a historical building could be recreated, with ancient artifacts appropriately placed inside.

[0768] Guide registration and management

[0769] Guides use a dedicated terminal application to enter their profile information (career history, expertise, achievements) and areas of expertise. This information is sent to the server and stored in a database. For example, a guide enters their profile as a historian, and that information is stored in the system.

[0770] User registration and matching system

[0771] Users create an account through the terminal application and enter their interests, desired exhibits, and available times. Based on this information, the server selects the most suitable guide and matches them with the user. For example, if a user expresses an interest in Renaissance paintings, the server will select a guide who is an expert in that field.

[0772] Tour execution and feedback collection

[0773] At the specified date and time, the guide and user access the application. The guide guides the user through the virtual museum and provides real-time commentary using communication software such as Zoom or Microsoft Teams. Users can share questions and impressions through chat functions or voice communication. After the tour, the user fills out a feedback form and sends it to the server. The server collects and analyzes this feedback and provides it to the guide and cultural facility. For example, while the guide explains about modern sculpture to the user, the user can ask questions freely and provide their impressions after the tour.

[0774] Revenue Share Distribution

[0775] The server collects the tour fee from the user and distributes it to the guide, the cultural facility, and the system operator. For example, the server automatically distributes the tour fee paid by credit card to each party.

[0776] ---

[0777] Example prompt sentence:

[0778] 1. "I'm interested in Japanese history. I'd like to see exhibits from the Kamakura period. Which guide would be best?"

[0779] 2. "I'd like to tour a virtual art museum next Saturday. Please select a guide who specializes in 17th-century European painting."

[0780] 3. "I'd like a detailed explanation of the Mona Lisa. Could you please check the digital archive of this exhibit and explain it to me?"

[0781] This system allows users to view the rich exhibits at cultural facilities from remote locations and receive real-time commentary from expert guides. It also enables two-way communication, allowing for feedback to be used to improve the quality of the service. Furthermore, the system's sustainability is ensured through appropriate distribution of fees.

[0782] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0783] Program processing flow and steps

[0784] Digital archiving of art museums

[0785] Step 1:

[0786] The server receives images, videos, and text data of exhibits from cultural facilities via an API.

[0787] Input: Exhibit data from cultural facilities (images, videos, text)

[0788] Output: The received data is loaded into the server's memory.

[0789] Specific operation: The server calls the API and receives and displays exhibit data in real time.

[0790] Step 2:

[0791] The server analyzes the received data and extracts metadata using image analysis techniques such as TensorFlow.

[0792] Input: Received exhibit data

[0793] Output: Metadata such as title, artist, era, description, etc.

[0794] Specific operation: The server executes the image analysis model and displays the analysis progress on the monitoring screen.

[0795] Step 3:

[0796] The server stores the analyzed metadata in a digital archive.

[0797] Input: Extracted metadata

[0798] Output: Metadata stored in a digital archive database

[0799] Specific behavior: The server performs the save operation and logs a save completion message.

[0800] Creating a virtual museum

[0801] Step 1:

[0802] The server retrieves the necessary data from the stored digital archive.

[0803] Input: Digital Archive Database

[0804] Output: Loaded exhibit data

[0805] What happens: The server executes the database query and loads the retrieved data into memory.

[0806] Step 2:

[0807] The server generates a three-dimensional virtual space using Unity or similar software.

[0808] Input: Loaded exhibit data

[0809] Output: Generated 3D virtual space

[0810] Specific operation: The server starts the Unity engine and displays the progress of the virtual space generation in real time.

[0811] Step 3:

[0812] The server appropriately arranges the exhibits in the virtual space.

[0813] Input: Exhibit data, 3D virtual space

[0814] Output: The completed virtual museum

[0815] Specific operation: The server executes the data placement algorithm and logs a placement completion message.

[0816] Guide registration and management

[0817] Step 1:

[0818] The terminal (guide) uses a dedicated application to input profile information and area of ​​expertise information.

[0819] Input: Guide profile information, specialty information

[0820] Output: The information entered is displayed on the terminal.

[0821] What happens: The application displays a confirmation message and checks the validity of the input data.

[0822] Step 2:

[0823] The terminal (guide) sends the entered information to the server.

[0824] Input: Profile information, area of ​​expertise

[0825] Output: Guide information uploaded to the server

[0826] Specific behavior: The application displays the progress of data transmission and notifies you when transmission is complete.

[0827] Step 3:

[0828] The server stores the received guide information in a database.

[0829] Input: Guide profile information, specialty information

[0830] Output: Guide information stored in a database

[0831] Specific behavior: The server performs the save operation and logs a save completion message.

[0832] User registration and matching system

[0833] Step 1:

[0834] The terminal (user) creates an account through the application.

[0835] Input: User account information

[0836] Output: A confirmation message is displayed to create an account.

[0837] What happens: The application checks the validity of the information entered and displays a confirmation message.

[0838] Step 2:

[0839] The terminal (user) inputs their interests and desired exhibit contents.

[0840] Input: User's interests and desired exhibits

[0841] Output: The information entered is displayed on the terminal.

[0842] What happens: The application displays a confirmation message and checks the validity of the input data.

[0843] Step 3:

[0844] The terminal (user) sends input information to the server.

[0845] Input: User's interests and desired exhibits

[0846] Output: User information uploaded to the server

[0847] Specific behavior: The application displays the progress of data transmission and notifies you when transmission is complete.

[0848] Step 4:

[0849] The server selects the most suitable guide based on the input guide information and user information and matches them with the user.

[0850] Input: Guide information, user information

[0851] Output: The matched guide-user pair

[0852] What happens: The server runs the matching algorithm and logs a match completion message.

[0853] Tour execution and feedback collection

[0854] Step 1:

[0855] At the specified date and time, the terminals (guide and user) access the application.

[0856] Input: User and guide access information

[0857] Output: Successfully logged into the application

[0858] Specific behavior: The application performs login authentication and displays a login completion message.

[0859] Step 2:

[0860] The device (guide) will guide you through the virtual museum and provide real-time commentary using Zoom, Microsoft Teams, etc.

[0861] Input: Guide explanation, user access information

[0862] Output: Real-time commentary, video and audio data

[0863] Specific operation: The guide provides commentary and broadcasts video and audio data in real time.

[0864] Step 3:

[0865] Devices (users) share questions and impressions through chat functions and voice communication.

[0866] Input: Questions and comments from users

[0867] Output: Questions and comments for the guide

[0868] What it does: The application forwards your questions and comments to the guide in real time.

[0869] Step 4:

[0870] After the tour ends, the terminal (user) fills in a feedback form and sends it to the server.

[0871] Input: Feedback information

[0872] Output: Feedback sent to the server

[0873] What happens: The application displays the progress of the feedback submission and notifies you when it is complete.

[0874] Step 5:

[0875] The server collects and analyzes the feedback.

[0876] Input: Feedback data

[0877] Output: Parsed feedback information

[0878] What happens: The server runs the analysis algorithm and logs the results.

[0879] Revenue Share Distribution

[0880] Step 1:

[0881] The server collects the tour fee from the user.

[0882] Input: User's payment information

[0883] Output: Fees collected

[0884] Specific behavior: The server collects the fee through the payment gateway and logs a collection completion message.

[0885] Step 2:

[0886] The server distributes the collected fees to guides, cultural facilities, and system operators.

[0887] Input: Collected fee data

[0888] Output: Distributed fees to each party

[0889] Specific operation: The server executes the distribution algorithm and sends a distribution completion message to each participant.

[0890] (Application example 1)

[0891] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0892] Conventional exhibit viewing systems at cultural facilities have not fully realized remote guided viewing, which has led to issues such as users being unable to effectively enjoy the exhibits using the latest technology. Furthermore, two-way communication, appropriate feedback collection, and matching of optimal guides based on user interests are also insufficient, hindering progress in the digitalization of cultural facilities.

[0893] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0894] In this invention, the server includes means for receiving exhibit data provided by the cultural facility and saving it as a digital archive, means for generating a three-dimensional virtual space based on the digital archive and appropriately arranging exhibits within the virtual space, means for inputting and managing guide profile information and specialty information, means for inputting and displaying user interests and desired exhibit content, means for matching the optimal guide with the user based on the input guide information and user information, means for the guide to guide the virtual space and provide real-time commentary, two-way communication means for users to submit questions and comments about the exhibits in real time via smartphones or tablet devices, means for collecting and analyzing feedback from users, means for providing the collected feedback to the guide and the cultural facility, and means for collecting tour fees from users and distributing the fees to the guide, the cultural facility, and the system operator. This enables the provision of high-quality cultural experiences remotely, matching users with the optimal guide based on their interests, providing real-time commentary and two-way communication, and analyzing and providing feedback.

[0895] A "cultural facility" is a place that displays exhibits and provides education and entertainment to the public, including art galleries, museums, and galleries.

[0896] "Exhibits" are objects such as artworks, historical materials, documents, etc. that are displayed in cultural institutions and that convey cultural or academic knowledge.

[0897] A "digital archive" is a system in which images, videos, and text data of exhibits are stored in digital format and managed as a database for future reference and search.

[0898] A "three-dimensional virtual space" is a virtual three-dimensional environment generated using computer technology, in which users can move freely within the space and visually appreciate exhibits and buildings that do not physically exist.

[0899] "Profile information" refers to data that includes information such as a guide's background, expertise, and track record, and this information is used to select and match guides.

[0900] "Specialty information" is information about a field or theme in which the guide is particularly knowledgeable, which enables the guide to provide specialized commentary and guidance to users.

[0901] "Matching" refers to the process of matching the most suitable guide with the user based on the user's interests and desired exhibits, as well as the guide's profile information and area of ​​expertise.

[0902] "Real-time commentary" means that a guide provides real-time audio and text explanations about exhibits and themes while guiding users through a virtual space, allowing users to obtain information instantly.

[0903] "Two-way communication" is a function that allows users to send questions and comments to guides and systems in real time via smartphones or tablet devices, engaging in two-way communication.

[0904] "Feedback" refers to the opinions and impressions that users enter after completing a tour, and this data is used to improve guides and cultural facilities.

[0905] "Fee collection" refers to the process of receiving tour fees from users through their payment method and distributing those fees to guides, cultural facilities, and system operators through the system.

[0906] A "generative AI model" is an artificial intelligence model that learns from large amounts of data to generate language and analyze data in a natural, human-like manner, and is used to analyze user feedback and select the optimal guide.

[0907] A "prompt" is an instruction given to a generative AI model to perform a specific task, allowing the AI ​​model to analyze and generate based on the specified content.

[0908] This invention is a system that enables users in remote locations to view exhibits at cultural facilities with a guide. The program for this system comprises the following steps:

[0909] The server receives data (images, videos, text) about exhibits provided by cultural facilities and stores it as a digital archive. Specifically, the server uses image analysis technology (such as OpenCV) to extract detailed information and stores it in a database (such as SQLite). This makes it possible to manage metadata (such as the name of the work, artist, era, and description) for each exhibit.

[0910] Next, the server generates a 3D virtual space based on the digital archive. Exhibits are placed in this virtual space in the same way as they would be in a real cultural facility. For example, the server uses a 3D rendering engine such as Unity or Unreal Engine to recreate a virtual art gallery or museum. The server also manages the placement data of the exhibits and places them in the appropriate locations.

[0911] Next, the guide enters and registers their profile information (career history, expertise, achievements) and areas of expertise into the server using a dedicated application. This saves and manages the guide information in a database. For example, if a guide enters their profile as a "historian," this information is saved in the system.

[0912] Users can create an account through the application and enter their interests and desired exhibits. They can also enter the time they are available. This information is sent to the server and analyzed. The server then matches users with the most suitable guide based on their interests. For example, if a user expresses an interest in "Renaissance paintings," the server will select a guide with that expertise.

[0913] At the designated date and time, the guide and user access the application. The guide will guide the user through the virtual space and provide real-time commentary. Users can send questions and comments about the exhibits in real time via their smartphones or tablets. This two-way communication function allows users to view the exhibits while communicating directly with the guide.

[0914] After the tour ends, feedback is collected from users. The server analyzes the collected feedback and provides it to guides and cultural facilities. The generative AI model also analyzes the feedback and suggests improvements to guide behavior and optimal placement of exhibits. Furthermore, by inputting prompts such as "Generate optimal guide suggestions based on the user's interests and feedback" into the generative AI model, the selection of the optimal guide can be automated.

[0915] Regarding fee collection, tour fees are collected from users and distributed to guides, cultural facilities, and system operators. For example, fees can be collected using credit cards or electronic payment services (such as PayPal), and the system automatically distributes them to each party.

[0916] As a specific example, if User A uses the app and inputs that he or she is interested in Renaissance paintings, the server will match the user with Guide B, who has specialized knowledge, based on that information. The user accesses the app at the specified date and time, and Guide B will guide the user through the virtual museum. After the tour ends, User A provides feedback, which is saved on the server. The generative AI model then analyzes the feedback and suggests areas for improving Guide B's explanation and optimal placement of exhibits.

[0917] Example prompt: "Generate optimal guide suggestions based on the user's interests and feedback."

[0918] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0919] Step 1: Receive exhibit data from cultural facilities and store it as a digital archive

[0920] The server receives images, videos, and text data of exhibits provided by cultural facilities via an API. This data is sent to the server, where image analysis technology (e.g., OpenCV) is used to extract detailed information (such as the name of the work, the artist, the year, and a description) and store it in a database (e.g., SQLite). The input is data about the exhibits, and the output is digitally archived data.

[0921] Step 2: Generate a 3D virtual space based on the digital archive

[0922] The server generates a virtual space using a 3D rendering engine such as Unity or Unreal Engine based on the stored digital archive. Exhibits are placed in appropriate positions within the virtual space. The input is digital archive data, and the output is a 3D virtual space.

[0923] Step 3: Enter and manage your guide's profile information and specialty information

[0924] The guide uses a terminal to input his / her profile information (career history, expertise, achievements) and areas of expertise. This information is sent to the server and stored in a database. The input is the guide's profile information, and the output is the guide information stored in the database.

[0925] Step 4: Enter your interests and desired exhibits

[0926] Users create an account using their device and enter their interests (the exhibits and themes they wish to view) and the time they have available. This information is sent to the server and analyzed. The input is the user's interests and desired exhibits, and the output is the analyzed user information.

[0927] Step 5: Matching the best guide with the user

[0928] The server performs appropriate matching based on the input guide information and user information. Using a generative AI model, it inputs a prompt such as "Generate optimal guide suggestions based on the user's interests and feedback" and selects the optimal guide. The input is guide information and user information, and the output is the matched guide and user.

[0929] Step 6: A guide will guide you through the virtual space

[0930] At the specified date and time, the guide and user access the application using their terminals. The guide will guide the user through the virtual space in real time and provide explanations of the exhibits. The input is the guide and user matched with the specified date and time, and the output is the real-time guided tour.

[0931] Step 7: Send your questions and comments via interactive communication

[0932] Visitors can send questions and comments about the exhibits in real time via smartphones or tablet devices, and the guide will respond accordingly. The input is the visitor's questions and comments, and the output is the guide's responses.

[0933] Step 8: Collect and analyze feedback

[0934] After the tour ends, feedback is collected from the user. The server analyzes the collected feedback and provides it to the guide and cultural facilities. It also analyzes the feedback using a generative AI model and proposes improvements to the guide's behavior and optimal placement of exhibits. The input is the user's feedback, and the output is the analysis results and proposals.

[0935] Step 9: Toll collection and distribution

[0936] The server collects tour fees from users and distributes them to guides, cultural facilities, and system operators using credit card payments or electronic payment services (such as PayPal). The input is the user's payment information, and the output is the distributed fee.

[0937] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0938] The system of the present invention allows remote users to view exhibits at cultural facilities with a guide, and also combines an emotion engine that recognizes the user's emotions. The system's program consists of many modules and methods, which provide users with a high-quality cultural experience and promote the improvement of guides and cultural facilities.

[0939] The main components of this system are as follows:

[0940] Digital archiving of art museums

[0941] server:

[0942] Images, videos, and text data of exhibits are received from cultural facilities via API. This data is stored on a server, and metadata (such as the name of the work, artist, year, and description) for each exhibit is analyzed and saved as a digital archive. For example, image data of a "famous painting" is received, and detailed information is extracted using image analysis technology and saved in the digital archive.

[0943] Creating a virtual museum

[0944] server:

[0945] The server generates a three-dimensional virtual space based on the stored digital archives. Within this virtual space, exhibits are arranged in the same way as they would be in a real cultural facility. For example, the interior of a "historical building" can be recreated, with "ancient artifacts" on display placed in appropriate locations.

[0946] Guide registration and management

[0947] Terminal (Guide):

[0948] Guides use a dedicated application to register their profile information (career history, expertise, achievements) and areas of expertise. This information is sent to the server and stored in a database. For example, a guide can enter a profile as a "historian," and that information is stored and managed within the system.

[0949] User registration and matching system

[0950] Terminal (user):

[0951] Users create an account through the application and enter their interests (the exhibits and themes they wish to view) and available time. The server matches them with the most suitable guide based on the information they enter. For example, if a user expresses an interest in "Renaissance paintings," the server will select a guide who is knowledgeable about Renaissance paintings.

[0952] Tour execution and feedback collection

[0953] Terminals (Guide and User):

[0954] At the specified date and time, the guide and user access the application. The guide guides the user through the virtual museum and provides real-time commentary. Users can share questions and comments through chat or voice communication. For example, while the guide explains "modern sculpture" to the user, the user can freely ask questions and provide feedback after the explanation.

[0955] server:

[0956] The server receives feedback from users and stores it in a database. The server analyzes the feedback and provides detailed reports to guides and cultural facilities.

[0957] Incorporating an emotion engine

[0958] Terminal (user):

[0959] The system uses an emotion engine that analyzes the user's facial expressions, tone of voice, etc. to monitor the user's emotional state in real time. For example, if the user feels surprise or excitement after seeing an exhibit, that emotional state will be analyzed.

[0960] server:

[0961] The emotion engine analyzes the user's emotional data and provides feedback to the guide based on that information, such as instructing the guide to provide a more detailed explanation of an exhibit that the user is interested in.

[0962] Revenue Share Distribution

[0963] server:

[0964] The server collects the tour fee from the user and distributes it to the guide, the cultural facility, and the system operator. For example, if the tour fee is paid by credit card, the server automatically distributes the fee to each party.

[0965] By integrating these functions, this system simultaneously digitizes cultural facilities, improves user convenience, and increases revenue. Furthermore, by incorporating an emotion engine, it can enrich the user experience and provide data useful for improving the operation of guides and cultural facilities. This is expected to promote financial support and digitalization for cultural facilities, and improve user cultural satisfaction.

[0966] The processing flow will be explained below.

[0967] Digital archiving of art museums

[0968] Step 1:

[0969] server:

[0970] Images, videos, and text data provided by cultural facilities are received via API.

[0971] Step 2:

[0972] server:

[0973] Store the received data in local storage or cloud storage.

[0974] Step 3:

[0975] server:

[0976] Metadata (title, artist, year, description) is extracted from the stored data using an image analysis algorithm.

[0977] Step 4:

[0978] server:

[0979] The extracted metadata is registered in a database.

[0980] Creating a virtual museum

[0981] Step 1:

[0982] server:

[0983] Using 3D modeling tools, a three-dimensional virtual space is generated based on the digital archive.

[0984] Step 2:

[0985] server:

[0986] The coordinate data of the exhibits is set within the generated virtual space.

[0987] Step 3:

[0988] server:

[0989] Based on the coordinate data, the exhibits are arranged and interactive functions are added.

[0990] Guide registration and management

[0991] Step 1:

[0992] Terminal (Guide):

[0993] The guide opens a dedicated application and enters their profile information and areas of expertise.

[0994] Step 2:

[0995] Terminal (Guide):

[0996] Add details about yourself, your achievements, etc.

[0997] Step 3:

[0998] server:

[0999] Receives guide input information and stores it in a database.

[1000] User registration and matching system

[1001] Step 1:

[1002] Terminal (user):

[1003] The user opens the dedicated application and creates an account.

[1004] Step 2:

[1005] Terminal (user):

[1006] Enter your interests, desired exhibits, and available time.

[1007] Step 3:

[1008] server:

[1009] It takes the user's input information and searches the database for the appropriate guide.

[1010] Step 4:

[1011] server:

[1012] Using an AI algorithm, it matches the user's interests, the guide's areas of expertise, and exhibit information.

[1013] Step 5:

[1014] server:

[1015] The matching results are presented to the user, and the user selects a guide.

[1016] Tour execution and feedback collection

[1017] Step 1:

[1018] Terminals (Guide and User):

[1019] The Guide and user access the application at the specified date and time.

[1020] Step 2:

[1021] Terminal (Guide):

[1022] A guide will guide you through the virtual museum and provide real-time commentary on the exhibits.

[1023] Step 3:

[1024] Terminal (user):

[1025] Users share questions and thoughts through chat functionality and voice communication.

[1026] Step 4:

[1027] Terminal (user):

[1028] After the tour, the user fills out a feedback form.

[1029] Step 5:

[1030] server:

[1031] Receive user feedback and store it in a database.

[1032] Step 6:

[1033] server:

[1034] The feedback data is analysed and provided to guides and cultural institutions as detailed reports.

[1035] Incorporating an emotion engine

[1036] Step 1:

[1037] Terminal (user):

[1038] Activate an emotion engine that analyzes the user's facial expressions and tone of voice.

[1039] Step 2:

[1040] Terminal (user):

[1041] The emotion engine analyzes the user's emotional state in real time and sends the data to the server.

[1042] Step 3:

[1043] server:

[1044] It receives emotional data sent in real time and generates feedback to the guide based on the analysis results.

[1045] Step 4:

[1046] Terminal (Guide):

[1047] The guide receives real-time feedback and adjusts the content and progress of the explanation based on the user's emotional state.

[1048] Revenue Share Distribution

[1049] Step 1:

[1050] server:

[1051] The tour fee is collected from the user via credit card or electronic payment.

[1052] Step 2:

[1053] server:

[1054] Temporarily store the collected fees.

[1055] Step 3:

[1056] server:

[1057] The fees will be distributed to guide, cultural institution and platform accounts based on a revenue share percentage.

[1058] Step 4:

[1059] server:

[1060] Process payments to each distribution recipient.

[1061] Through these processing steps, the system of the present invention simultaneously digitizes cultural facilities, improves user convenience, and increases revenue. Furthermore, by incorporating an emotion engine, it can enrich the user experience and provide data that is useful for improving the operation of guides and cultural facilities.

[1062] Example 2

[1063] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1064] When remote visitors view exhibits at cultural facilities, they need a high-quality, immersive experience while also being able to effectively collect, manage, and utilize their feedback on guides and cultural facilities. A system is also needed that can grasp the emotional state of visitors in real time and enable guides to respond appropriately based on that information. Fair and efficient distribution of revenue is also required.

[1065] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1066] In this invention, the server includes means for receiving exhibit data provided by the cultural facility and saving it as a digital archive, means for generating a three-dimensional virtual space based on the digital archive and appropriately arranging the exhibits within the virtual space, and means for inputting and managing guide profile information and specialty information. This allows remote users to enjoy a high-quality, immersive cultural experience while also enabling monitoring of the users' emotional states and real-time responses, and realizing fair and efficient distribution of revenue.

[1067] "Exhibits" refers to artworks, historical artifacts, artifacts, and other cultural assets that are made available to the public at cultural institutions.

[1068] A "guide" refers to a person who has knowledge of the exhibits at a cultural facility and provides explanations to visitors.

[1069] "User" refers to an individual in a remote location who wishes to view the exhibits at a cultural facility.

[1070] A "digital archive" refers to a system that stores and manages exhibit data (images, videos, text, etc.) in digital format.

[1071] "Three-dimensional virtual space" refers to a three-dimensional virtual space generated using digital technology that recreates the interior of an actual cultural facility.

[1072] "Profile information" refers to information about the guide, such as their background, expertise, and achievements.

[1073] "Emotional state" refers to emotions (surprise, excitement, joy, etc.) analyzed from the user's facial expressions and tone of voice.

[1074] "Feedback" refers to impressions, opinions, and ratings collected from users regarding exhibits and guides.

[1075] "Distribution" refers to the fair and efficient division of revenues (e.g., tour fees) among guides, cultural institutions, and system operators.

[1076] The system of the present invention allows remote users to view exhibits at cultural facilities with a guide, and also combines an emotion engine that recognizes the user's emotions. The system is composed of multiple modules and means, providing users with a high-quality cultural experience and promoting the improvement of guides and cultural facilities.

[1077] Digital archiving of art museums

[1078] server:

[1079] Images, videos, and text data of exhibits are received from cultural facilities via API. For example, TensorFlow is used as a specific image analysis technology to extract the name of the work and information about the artist from the received image data. This data is stored on a server as metadata (work name, artist, year, description, etc.) and managed as a digital archive. As a specific example, image data of a famous painting is received, and the name of the work and information about the artist are extracted using image analysis technology.

[1080] Creating a virtual museum

[1081] server:

[1082] A 3D virtual space is generated based on the stored digital archives. 3D models of the exhibits are created using 3D engines such as Unity or Unreal Engine, and then placed inside the virtual museum. Specifically, the interior of a "historical building" is recreated, and the "ancient artifacts" to be displayed there are placed in the appropriate locations.

[1083] Guide registration and management

[1084] Terminal (navigator):

[1085] Guides use a dedicated application to register their profile information (career history, expertise, achievements) and areas of expertise. This information is sent to the server and stored in a database. For example, if a guide enters their profile as a "historian," the information is sent to the server in real time and stored in the database.

[1086] User registration and matching system

[1087] Terminal (user):

[1088] Users create an account through the application and input their interests (the exhibits and themes they wish to view) and available time. The server matches them with the most suitable guide based on the information they input. For example, if a user expresses an interest in "Renaissance paintings," the server will select a guide who is knowledgeable about Renaissance paintings.

[1089] Tour execution and feedback collection

[1090] Terminals (instructor and user):

[1091] At the designated date and time, the guide and user access the application. The guide will guide the user through the virtual museum and provide real-time commentary. Users can share their questions and impressions through chat and voice communication. For example, while the guide is explaining about "modern sculpture," users can freely ask questions and provide feedback after the explanation.

[1092] server:

[1093] Feedback from users is received and stored in a database. The server analyzes the feedback and provides detailed reports to guides and cultural facilities.

[1094] Incorporating an emotion engine

[1095] Terminal (user):

[1096] The system uses an emotion engine that analyzes the user's facial expressions and tone of voice to monitor the user's emotional state in real time. For example, the system uses the emotion recognition API from Microsoft Azure Cognitive Services as a specific analysis technology. For example, if a user feels surprise or excitement when looking at an exhibit, that emotional state is analyzed.

[1097] server:

[1098] The emotion engine analyzes the user's emotional data and provides feedback to the guide based on that information. For example, it can instruct the guide to provide a more detailed explanation of an exhibit that the user is interested in.

[1099] Revenue Share Distribution

[1100] server:

[1101] The server collects the tour fee from the user and distributes it to the guide, the cultural facility, and the system operator. Specifically, it uses a payment gateway API such as Stripe for payment processing. When the tour fee is paid by credit card, the server automatically distributes the fee to each party.

[1102] Specific examples and examples of prompts for generative AI models

[1103] As a concrete example, consider a case where a user wants to view a "famous Renaissance painting." In this case, the server selects a "guide with expertise in Renaissance painting." If the emotion engine detects the user's excitement, it instructs the guide to provide more detailed information about the exhibit. An example of a prompt sentence for the generative AI model is as follows:

[1104] 1. "User is interested in Renaissance paintings. Please select relevant guides."

[1105] 2. "The user seems surprised by the exhibit. Please instruct the guide to provide a detailed explanation."

[1106] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1107] Step 1:

[1108] The server receives and stores data from the cultural institution.

[1109] Input: Images, videos, and text data of exhibits provided by cultural facilities via API

[1110] Specific operation: The server receives data on the exhibits using an API. To do this, it uses image analysis technology such as TensorFlow to extract the name of the work and information about the artist from the received image data and saves it as metadata.

[1111] Output: The analyzed metadata and digital data of the exhibits are saved on the server.

[1112] Step 2:

[1113] The server generates a three-dimensional virtual space.

[1114] Input: Stored digital archive data

[1115] How it works: The server uses a 3D engine such as Unity or Unreal Engine to generate a 3D virtual space based on the stored digital archive data. 3D models of the exhibits are generated and placed appropriately within the virtual museum.

[1116] Output: A 3D virtual museum space accessible to visitors and guides

[1117] Step 3:

[1118] The device (guide) registers and manages profile information

[1119] Input: Guide's profile information (career history, expertise, achievements) and area of ​​expertise

[1120] Specific operation: The guide uses a dedicated application to enter his / her profile information and send it to the server. This information is saved in real time on the server and managed in a database.

[1121] Output: Guide profile information stored on the server

[1122] Step 4:

[1123] The device (user) inputs interest information

[1124] Input: User's interests (exhibits and themes they wish to see) and available time input through the application

[1125] How it works: A user uses the application to input the exhibits and themes they wish to see, as well as the time they have available. This information is sent to the server and stored in a database.

[1126] Output: User interest information and available time stored on the server

[1127] Step 5:

[1128] The server matches guides with users.

[1129] Input: Guide profile information and user interest information stored in the database

[1130] How it works: The server collates the user's interests and automatically selects the most suitable guide. This process may involve the use of generative AI models. For example, a prompt might read, "The user is interested in Renaissance paintings. Please select a relevant guide."

[1131] Output: A properly matched guide-user pair

[1132] Step 6:

[1133] Conducting guided tours and providing real-time commentary

[1134] Input: Pre-matched guide and user, 3D virtual space data

[1135] Specific operation: At the specified date and time, the guide and user access the application and begin a tour of the virtual museum. The guide explains the exhibits in real time, and users share their questions and impressions using the chat function and voice communication.

[1136] Output: Real-time commentary and user questions and comments

[1137] Step 7:

[1138] The server collects and analyzes the feedback

[1139] Input: User feedback (chat messages, impressions, ratings)

[1140] Specific operation: After the tour ends, the server collects feedback from users and stores it in a database. It then analyzes the feedback and provides a detailed report to the guide and the cultural facility.

[1141] Output: Parsed feedback report

[1142] Step 8:

[1143] Monitor emotional states using the Emotion Engine

[1144] Input: Real-time user facial expressions and tone of voice

[1145] Specific behavior: Using an emotion engine (e.g., Microsoft Azure Cognitive Services), the user's facial expressions and tone of voice are analyzed in real time to monitor their emotional state. The analysis results are sent to the server. An example prompt might be, "The user seems surprised by the exhibits. Please instruct the guide to provide a detailed explanation."

[1146] Output: Real-time analyzed emotion data

[1147] Step 9:

[1148] The server provides feedback based on emotion data

[1149] Input: Parsed emotion data

[1150] Specific operation: The server provides real-time feedback to the guide based on the analyzed emotion data, allowing the guide to provide more detailed information about the exhibits that the user is interested in.

[1151] Output: Feedback instructions based on emotion data

[1152] Step 10:

[1153] The server collects and distributes the tour fee.

[1154] Input: Tour payment information from user

[1155] Specific operation: The server uses a payment gateway API such as Stripe to collect tour fees from users, and distributes the collected fees fairly and efficiently among guides, cultural facilities, and system operators.

[1156] Output: Tour fare distributed to each party

[1157] (Application example 2)

[1158] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1159] Conventional systems for remote viewing of exhibits at cultural facilities have had the problem of being unable to provide a personalized experience based on the user's emotions and interests. In addition, they lacked the functionality for users to obtain information interactively about the exhibits, which could result in a decrease in the user's understanding of the explanations and a decrease in their satisfaction.

[1160] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for receiving exhibit data provided by a cultural facility and saving it as a digital archive; means for generating a three-dimensional virtual space based on the digital archive and appropriately arranging exhibits in the virtual space; means for inputting and managing guide profile information and specialty information; means for inputting and displaying the user's interests and desired exhibit content; means for matching the optimal guide with the user based on the input guide information and user information; means for the guide to guide the virtual space and provide commentary in real time; means for collecting and analyzing feedback from the user; means for providing the collected feedback to the guide and the cultural facility; means for collecting tour fees from the user and distributing the fees to the guide, the cultural facility, and the system operator; and means for analyzing the user's emotions and adjusting the guide's commentary based on the analysis data. This allows users to obtain information about the exhibits interactively and provides a personalized experience tailored to the user's emotions and interests.

[1161] "Cultural facilities" are facilities that collect, preserve, and display cultural exhibits and materials, such as art galleries, museums, and libraries.

[1162] An "exhibit" is a tangible item, such as a work of art, historical artifact, scientific specimen, or document, that is displayed to visitors at a cultural institution.

[1163] A "digital archive" is an electronic database for storing and managing images, videos, text data, etc. of exhibits in digital format.

[1164] A "three-dimensional virtual space" is a three-dimensional virtual space generated by a computer, and is a virtual environment that a user can experience using a virtual reality device or the like.

[1165] A "guide" is a person or system that has specialized knowledge about cultural facilities and exhibits and provides explanations and guidance to users.

[1166] "Profile Information" refers to personal information about a guide, such as their background, expertise, achievements, and areas of expertise.

[1167] "Interest" refers to the degree of interest a visitor has in exhibits or themes that they find particularly appealing.

[1168] "Matching" is the process of selecting the most suitable guide based on the user's interests and desired exhibits, and connecting the user with the guide.

[1169] "Feedback" refers to reactions such as impressions, evaluations, and opinions from users, and is the process of collecting and analyzing these to help improve services.

[1170] "Emotion analysis" is a technology that analyzes a user's facial expressions and tone of voice to assess their emotional state in real time.

[1171] "Personalization" means individually tailoring and providing services and content to suit each user's interests and emotions.

[1172] An embodiment of the present invention provides a remote guided viewing experience of exhibits at a cultural facility by linking a server, terminals (guides, users), and an emotion analysis system. Specific embodiments are described below.

[1173] 1. Digital Archiving

[1174] The server receives images, videos, and text data of exhibits provided by cultural facilities via an API. This data is stored in a database on the server, and metadata (such as the title of the work, artist, year, and description) for each exhibit is analyzed and saved as a digital archive. Specifically, a cloud server on AWS (Amazon Web Services) is used, and the saved data is managed in an SQL database.

[1175] 2. Creating a virtual store

[1176] The server generates a 3D virtual space based on the stored digital archives. Unity and Unreal Engine are used to appropriately position exhibits within this virtual space. This virtually recreates the layout of the actual cultural facility, providing users with a realistic experience.

[1177] 3. Guide registration and management

[1178] Guides use a dedicated application to enter their profile information (career history, expertise, achievements) and areas of expertise, and then send this information to a server. This data is stored in a database on the server and later used to match users. The device can be a smartphone, tablet, or PC.

[1179] 4. User Registration and Matching System

[1180] Users create an account through the application and enter their interests (exhibits and themes they wish to see) and available times. The server analyzes this information and matches them with the most suitable guide. In particular, frameworks such as React and AngularJS are used for the user interface.

[1181] 5. Product Tour and Feedback Collection

[1182] At a specified date and time, the guide and user access the application. The guide navigates the virtual space and provides commentary in real time. Users share questions and impressions through chat and voice communication. Commentary is provided using a combination of speech recognition and text conversion systems, and feedback is sent to a server for storage. Python and TensorFlow are used for feedback analysis.

[1183] 6. Incorporating an Emotion Analysis System

[1184] An emotion engine that analyzes the user's facial expressions and tone of voice is installed on the device to monitor the user's emotional state in real time. This is done using OpenCV and Azure Emotion API. Emotion data is sent to a server, which provides feedback to the guide based on the analysis results.

[1185] 7. Revenue Share Distribution

[1186] The server collects the tour fee from the user and automatically distributes it to the guide, the cultural facility, and the system operator. Payment is processed using electronic payment services such as Stripe and PayPal.

[1187] Specific examples

[1188] For example, if a user expresses interest in a "historical building" exhibit in a virtual store, the system will use that information to match with a guide knowledgeable in history and conduct a tour. If the user shows excitement at an "ancient relic" displayed during the tour, the emotion engine will analyze this emotion and prompt the guide to provide a detailed explanation.

[1189] Example prompts for generative AI models

[1190] "Analyze how users are responding to this product. Based on image and audio data, determine whether their reactions are positive or negative based on facial expressions and tone of voice, and reflect this in a detailed guide."

[1191] This allows visitors to interact with the exhibits and obtain information about them, and also enables a personalized experience based on the visitors' emotions and interests.

[1192] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1193] Step 1:

[1194] The server receives images, videos, and text data of exhibits provided by cultural facilities via API. The input data is the exhibit's metadata (work title, artist, year, description, etc.), which is stored in a database. The server analyzes the stored data and organizes it into a digital archive. Specifically, it performs an INSERT operation on the database table, saves the data, and generates a CONFIRM message.

[1195] Step 2:

[1196] The server generates a three-dimensional virtual space using Unity or Unreal Engine based on the saved digital archive. The input data is the digital archive data saved in step 1. This also includes location and layout information for the exhibits to be placed appropriately within the virtual space. Specifically, the server uses Unity's Scene Builder function to place the 3D models of the exhibits within the scene and generate EXPORTED scene data.

[1197] Step 3:

[1198] The terminal (guide) uses a dedicated application to input their own profile information (career history, expertise, achievements) and areas of expertise. The input data is the guide's personal information and expertise information, and is sent to the server. The server stores the sent data in a database. Specifically, the terminal receives the input data through the GUI, sends a POST request to the server, and the server inserts it into the database.

[1199] Step 4:

[1200] The device (user) creates an account through the application and inputs their interests (exhibits and themes they wish to view) and available times. The input data is the user's personal information and interest data, and is sent to the server. The server analyzes this and matches them with the most suitable guide. Specifically, the server applies guide information and a matching algorithm based on the user's interest data to generate MATCHED parameters.

[1201] Step 5:

[1202] At the specified date and time, the guide and user access the application. The guide navigates the virtual space and provides commentary in real time. The input data is the scene data of the virtual space and the commentary, which is displayed and played on the device. Specifically, the device renders the virtual space data and displays the guide's commentary in audio or text.

[1203] Step 6:

[1204] An emotion engine that analyzes the user's facial expressions and tone of voice is installed on the device, monitoring the user's emotional state in real time. The input data is images of the user's facial expressions and voice data, which is analyzed to generate emotional state data. Specifically, the device acquires data from the camera and microphone, runs an emotion analysis algorithm (OpenCV or Azure Emotion API), and sends the analysis results to the server.

[1205] Step 7:

[1206] The server provides feedback to the guide based on the emotion analysis results sent by the user. The input data is emotional state data, and based on this, feedback data is generated to adjust the guide's explanations. Specifically, the server analyzes the emotion data, generates a feedback message instructing the necessary action (strengthening the explanation, providing more detailed explanations, etc.), and sends a push notification to the guide's device.

[1207] Step 8:

[1208] After the tour ends, the server collects feedback from users and stores it in a database. The input data is the user's feedback information, which is analyzed to generate report data to be provided to cultural facilities and guides. Specifically, the server receives the feedback information, applies an analytical algorithm (such as natural language processing) to create a report, and sends it to the relevant parties.

[1209] Step 9:

[1210] The server collects tour fees from users and automatically distributes them to guides, cultural facilities, and system operators. The input data is payment information and distribution ratio data, and distribution transactions are carried out based on this. Specifically, the server collects fees through a payment gateway (Stripe or PayPal) and automatically makes payments to the parties involved based on the pre-set distribution ratio.

[1211] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1212] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1213] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1214] [Third embodiment]

[1215] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1216] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1217] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1218] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1219] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1220] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1221] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1222] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1223] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1224] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1225] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1226] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1227] The system of the present invention allows users to view exhibits at cultural facilities remotely and provides guided tours. The system's program consists of numerous modules and tools, providing users with a high-quality cultural experience.

[1228] The main components of this system are as follows:

[1229] Digital archiving of art museums

[1230] server:

[1231] Images, videos, and text data of exhibits are received from cultural facilities via API. This data is stored on the server and analyzed to register metadata (title, artist, era, description, etc.) for each exhibit. As a specific example, the server receives image data of a "famous painting," extracts detailed information about it using image analysis technology, and stores it in a digital archive.

[1232] Creating a virtual museum

[1233] server:

[1234] The server generates a three-dimensional virtual space based on the stored digital archives. Within this virtual space, exhibits are arranged in the same way as they would be in a real cultural facility. For example, the server recreates the interior of a "historical building" and places the "ancient artifacts" on display in the appropriate locations.

[1235] Guide registration and management

[1236] Terminal (Guide):

[1237] Guides use a dedicated application to register their profile information (career history, expertise, achievements) and areas of expertise. This information is sent to the server and stored in a database. For example, a guide can enter a profile as a "historian," and that information is stored and managed within the system.

[1238] User registration and matching system

[1239] Terminal (user):

[1240] Users create an account through the application and enter their interests (the exhibits and themes they wish to view) and available time. The server matches them with the most suitable guide based on the information they enter. For example, if a user expresses an interest in "Renaissance paintings," the server will select a guide who is knowledgeable about Renaissance paintings.

[1241] Tour execution and feedback collection

[1242] Terminals (Guide and User):

[1243] At the specified date and time, the guide and user access the application. The guide guides the user through the virtual museum and provides commentary in real time. Users can share questions and impressions through chat or voice communication. After the tour ends, the user fills out a feedback form and sends it to the server. The server collects and analyzes the feedback and provides it to the guide and cultural facility. For example, while the guide explains to the user about "modern sculpture," the user is free to ask questions and provide feedback after the tour ends.

[1244] Revenue Share Distribution

[1245] server:

[1246] The server collects the tour fee from the user and distributes it to the guide, the cultural facility, and the system operator. For example, if the tour fee is paid by credit card, the server automatically distributes the fee to each party.

[1247] By integrating these functions, this system will solve the problems of cultural facilities, such as a lack of funding, delays in digitalization, and difficulty in accessing the system for users, and improve the quality of the cultural experience. Each function has been explained with concrete examples, and it is expected that this will increase the revenue of cultural facilities and improve user satisfaction.

[1248] The processing flow will be explained below.

[1249] Digital archiving of art museums

[1250] Step 1:

[1251] server:

[1252] Images, videos, and text data provided by cultural facilities are received via API.

[1253] Step 2:

[1254] server:

[1255] Store the received data in local storage or cloud storage.

[1256] Step 3:

[1257] server:

[1258] Metadata (title, artist, year, description) is extracted from the stored data using an image analysis algorithm.

[1259] Step 4:

[1260] server:

[1261] The extracted metadata is registered in a database.

[1262] Creating a virtual museum

[1263] Step 1:

[1264] server:

[1265] Using 3D modeling tools, a three-dimensional virtual space is generated based on the digital archive.

[1266] Step 2:

[1267] server:

[1268] The coordinate data of the exhibits is set within the generated virtual space.

[1269] Step 3:

[1270] server:

[1271] Based on the coordinate data, the exhibits are arranged and interactive functions are added.

[1272] Guide registration and management

[1273] Step 1:

[1274] Terminal (Guide):

[1275] The guide opens a dedicated application and enters their profile information and areas of expertise.

[1276] Step 2:

[1277] Terminal (Guide):

[1278] Also add detailed information such as your self-introduction and achievements.

[1279] Step 3:

[1280] server:

[1281] Receives guide input information and stores it in a database.

[1282] User registration and matching system

[1283] Step 1:

[1284] Terminal (user):

[1285] The user opens the dedicated application and creates an account.

[1286] Step 2:

[1287] Terminal (user):

[1288] Enter your interests, desired exhibits, and available time.

[1289] Step 3:

[1290] server:

[1291] It takes the user's input information and searches the database for the appropriate guide.

[1292] Step 4:

[1293] server:

[1294] Using an AI algorithm, it matches the user's interests, the guide's areas of expertise, and exhibit information.

[1295] Step 5:

[1296] server:

[1297] The matching results are presented to the user, and the user selects a guide.

[1298] Tour execution and feedback collection

[1299] Step 1:

[1300] Terminals (Guide and User):

[1301] The Guide and user access the application at the specified date and time.

[1302] Step 2:

[1303] Terminal (Guide):

[1304] A guide will guide you through the virtual museum and provide real-time commentary on the exhibits.

[1305] Step 3:

[1306] Terminal (user):

[1307] Users share questions and thoughts through chat functionality and voice communication.

[1308] Step 4:

[1309] Terminal (user):

[1310] After the tour, the user fills out a feedback form.

[1311] Step 5:

[1312] server:

[1313] Receive user feedback and store it in a database.

[1314] Step 6:

[1315] server:

[1316] The feedback data is analysed and provided to guides and cultural institutions as detailed reports.

[1317] Revenue Share Distribution

[1318] Step 1:

[1319] server:

[1320] The tour fee is collected from the user via credit card or electronic payment.

[1321] Step 2:

[1322] server:

[1323] Temporarily store the collected fees.

[1324] Step 3:

[1325] server:

[1326] The fees will be distributed to guide, cultural institution and platform accounts based on a revenue share percentage.

[1327] Step 4:

[1328] server:

[1329] Process payments to each distribution recipient.

[1330] Through these processing steps, the system of the present invention simultaneously realizes the digitization of cultural facilities, improved convenience for users, and increased profits.

[1331] Example 1

[1332] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1333] There is a need to provide the experience of viewing exhibits at cultural facilities to users in remote locations. Another challenge is to enable users to not only receive real-time commentary from guides, but also to select guides that suit their interests. Furthermore, it is necessary to collect and analyze user feedback and provide it to guides and cultural facilities to improve the quality of services. Finally, to ensure the economic sustainability of this system, an appropriate distribution of revenue shares must be addressed.

[1334] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1335] In this invention, the server includes: means for receiving exhibit data provided by the cultural facility and saving it as a digital archive; means for generating a three-dimensional virtual space based on the digital archive and appropriately arranging exhibits within the virtual space; means for inputting and managing guide profile information and specialty information; means for inputting and displaying user interests and desired exhibits; means for matching the optimal guide with the user based on the input guide information and user information; means for the guide and user to access the application at a specified date and time and provide real-time commentary; interactive means for users to share questions and impressions via chat or voice communication; means for collecting and analyzing feedback from users; means for providing the collected feedback to the guide and the cultural facility; and means for collecting tour fees from users and distributing the fees to the guide, the cultural facility, and the system operator. This allows users to view exhibits at the cultural facility in real time, even from remote locations, and enjoy an interactive experience tailored to their interests while listening to the guide's commentary. Furthermore, collecting and analyzing feedback and appropriately distributing fees improves the quality of the system and ensures its economic sustainability.

[1336] A "remotely located user" is a user who is physically located away from the cultural facility but who views the exhibits at the cultural facility through this system.

[1337] "Cultural facilities" refer to facilities that store and display exhibits, such as art galleries and museums.

[1338] "Exhibits" are objects or materials that are available for viewing, such as works of art, archaeological artifacts, or natural history exhibits, that are displayed at cultural institutions.

[1339] A "digital archive" is a database for digitally storing and managing data, including images, videos, and text of exhibits provided by cultural facilities.

[1340] A "three-dimensional virtual space" is a three-dimensional digital space generated by a computer, a virtual exhibition space that users can access and experience via the Internet.

[1341] A "guide" is a person with specialized knowledge who provides explanations and commentary on exhibits to visitors and leads tours.

[1342] "Profile information" refers to information about the guide's qualifications and abilities, such as the guide's background, expertise, and achievements.

[1343] "Specialty information" refers to information about areas or topics in which the guide is particularly knowledgeable.

[1344] "Interests" refer to the exhibits or themes that visitors are particularly interested in viewing.

[1345] "Desired exhibit content" refers to the specific exhibits or themes that the user would like to see on the tour.

[1346] "Matching" refers to the process of matching the most suitable guide with the user based on the guide information and user information entered.

[1347] "Application" refers to the software that allows guides and users to access the system and perform various operations.

[1348] "Interactive function" refers to a function that allows two-way communication, such as allowing users to ask questions about exhibits and obtain information in real time.

[1349] "Feedback" refers to impressions, evaluations, suggestions for improvement, and other information provided by users after the tour has ended.

[1350] "Feedback collection" refers to the process of collecting feedback provided by users as data.

[1351] "Feedback analysis" refers to the process of analyzing collected feedback to help improve the quality of services.

[1352] "Tour Fee" means the fee paid by a User to participate in a Guided Tour.

[1353] "Revenue share" is a system in which collected tour fees are distributed to guides, cultural facilities, and system operators.

[1354] MODE FOR CARRYING OUT THE INVENTION

[1355] The system of the present invention allows users in remote locations to view exhibits at cultural facilities and experience guided tours. This system operates in cooperation with a server, terminals, and users. A specific implementation method is described below.

[1356] Digital archiving of art museums

[1357] The server first receives data (images, videos, text, etc.) about exhibits from cultural facilities via an API. The received data is analyzed using image analysis software such as TensorFlow, and metadata such as the work's name, artist, era, and description is extracted. This data is then stored as a digital archive. For example, the server could receive image data of a "famous painting," extract its detailed information, and store it in a digital archive.

[1358] Creating a virtual museum

[1359] The server uses the stored digital archives to create a virtual space using 3D virtual space generation software such as Unity. Exhibits are placed in this virtual space just as they would be in a real cultural facility. For example, a historical building could be recreated, with ancient artifacts appropriately placed inside.

[1360] Guide registration and management

[1361] Guides use a dedicated terminal application to enter their profile information (career history, expertise, achievements) and areas of expertise. This information is sent to the server and stored in a database. For example, a guide enters their profile as a historian, and that information is stored in the system.

[1362] User registration and matching system

[1363] Users create an account through the terminal application and enter their interests, desired exhibits, and available times. Based on this information, the server selects the most suitable guide and matches them with the user. For example, if a user expresses an interest in Renaissance paintings, the server will select a guide who is an expert in that field.

[1364] Tour execution and feedback collection

[1365] At the specified date and time, the guide and user access the application. The guide guides the user through the virtual museum and provides real-time commentary using communication software such as Zoom or Microsoft Teams. Users can share questions and impressions through chat functions or voice communication. After the tour, the user fills out a feedback form and sends it to the server. The server collects and analyzes this feedback and provides it to the guide and cultural facility. For example, while the guide explains about modern sculpture to the user, the user can ask questions freely and provide their impressions after the tour.

[1366] Revenue Share Distribution

[1367] The server collects the tour fee from the user and distributes it to the guide, the cultural facility, and the system operator. For example, the server automatically distributes the tour fee paid by credit card to each party.

[1368] ---

[1369] Example prompt sentence:

[1370] 1. "I'm interested in Japanese history. I'd like to see exhibits from the Kamakura period. Which guide would be best?"

[1371] 2. "I'd like to tour a virtual art museum next Saturday. Please select a guide who specializes in 17th-century European painting."

[1372] 3. "I'd like a detailed explanation of the Mona Lisa. Could you please check the digital archive of this exhibit and explain it to me?"

[1373] This system allows users to view the rich exhibits at cultural facilities from remote locations and receive real-time commentary from expert guides. It also enables two-way communication, allowing for feedback to be used to improve the quality of the service. Furthermore, the system's sustainability is ensured through appropriate distribution of fees.

[1374] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1375] Program processing flow and steps

[1376] Digital archiving of art museums

[1377] Step 1:

[1378] The server receives images, videos, and text data of exhibits from cultural facilities via an API.

[1379] Input: Exhibit data from cultural facilities (images, videos, text)

[1380] Output: The received data is loaded into the server's memory.

[1381] Specific operation: The server calls the API and receives and displays exhibit data in real time.

[1382] Step 2:

[1383] The server analyzes the received data and extracts metadata using image analysis techniques such as TensorFlow.

[1384] Input: Received exhibit data

[1385] Output: Metadata such as title, artist, era, description, etc.

[1386] Specific operation: The server executes the image analysis model and displays the analysis progress on the monitoring screen.

[1387] Step 3:

[1388] The server stores the analyzed metadata in a digital archive.

[1389] Input: Extracted metadata

[1390] Output: Metadata stored in a digital archive database

[1391] Specific behavior: The server performs the save operation and logs a save completion message.

[1392] Creating a virtual museum

[1393] Step 1:

[1394] The server retrieves the necessary data from the stored digital archive.

[1395] Input: Digital Archive Database

[1396] Output: Loaded exhibit data

[1397] What happens: The server executes the database query and loads the retrieved data into memory.

[1398] Step 2:

[1399] The server generates a three-dimensional virtual space using Unity or similar software.

[1400] Input: Loaded exhibit data

[1401] Output: Generated 3D virtual space

[1402] Specific operation: The server starts the Unity engine and displays the progress of the virtual space generation in real time.

[1403] Step 3:

[1404] The server appropriately arranges the exhibits in the virtual space.

[1405] Input: Exhibit data, 3D virtual space

[1406] Output: The completed virtual museum

[1407] Specific operation: The server executes the data placement algorithm and logs a placement completion message.

[1408] Guide registration and management

[1409] Step 1:

[1410] The terminal (guide) uses a dedicated application to input profile information and area of ​​expertise information.

[1411] Input: Guide profile information, specialty information

[1412] Output: The information entered is displayed on the terminal.

[1413] What happens: The application displays a confirmation message and checks the validity of the input data.

[1414] Step 2:

[1415] The terminal (guide) sends the entered information to the server.

[1416] Input: Profile information, area of ​​expertise

[1417] Output: Guide information uploaded to the server

[1418] Specific behavior: The application displays the progress of data transmission and notifies you when transmission is complete.

[1419] Step 3:

[1420] The server stores the received guide information in a database.

[1421] Input: Guide profile information, specialty information

[1422] Output: Guide information stored in a database

[1423] Specific behavior: The server performs the save operation and logs a save completion message.

[1424] User registration and matching system

[1425] Step 1:

[1426] The terminal (user) creates an account through the application.

[1427] Input: User account information

[1428] Output: A confirmation message is displayed to create an account.

[1429] What happens: The application checks the validity of the information entered and displays a confirmation message.

[1430] Step 2:

[1431] The terminal (user) inputs their interests and desired exhibit contents.

[1432] Input: User's interests and desired exhibits

[1433] Output: The information entered is displayed on the terminal.

[1434] What happens: The application displays a confirmation message and checks the validity of the input data.

[1435] Step 3:

[1436] The terminal (user) sends input information to the server.

[1437] Input: User's interests and desired exhibits

[1438] Output: User information uploaded to the server

[1439] Specific behavior: The application displays the progress of data transmission and notifies you when transmission is complete.

[1440] Step 4:

[1441] The server selects the most suitable guide based on the input guide information and user information and matches them with the user.

[1442] Input: Guide information, user information

[1443] Output: The matched guide-user pair

[1444] What happens: The server runs the matching algorithm and logs a match completion message.

[1445] Tour execution and feedback collection

[1446] Step 1:

[1447] At the specified date and time, the terminals (guide and user) access the application.

[1448] Input: User and guide access information

[1449] Output: Successfully logged into the application

[1450] Specific behavior: The application performs login authentication and displays a login completion message.

[1451] Step 2:

[1452] The device (guide) will guide you through the virtual museum and provide real-time commentary using Zoom, Microsoft Teams, etc.

[1453] Input: Guide explanation, user access information

[1454] Output: Real-time commentary, video and audio data

[1455] Specific operation: The guide provides commentary and broadcasts video and audio data in real time.

[1456] Step 3:

[1457] Devices (users) share questions and impressions through chat functions and voice communication.

[1458] Input: Questions and comments from users

[1459] Output: Questions and comments for the guide

[1460] What it does: The application forwards your questions and comments to the guide in real time.

[1461] Step 4:

[1462] After the tour ends, the terminal (user) fills in a feedback form and sends it to the server.

[1463] Input: Feedback information

[1464] Output: Feedback sent to the server

[1465] What happens: The application displays the progress of the feedback submission and notifies you when it is complete.

[1466] Step 5:

[1467] The server collects and analyzes the feedback.

[1468] Input: Feedback data

[1469] Output: Parsed feedback information

[1470] What happens: The server runs the analysis algorithm and logs the results.

[1471] Revenue Share Distribution

[1472] Step 1:

[1473] The server collects the tour fee from the user.

[1474] Input: User's payment information

[1475] Output: Fees collected

[1476] Specific behavior: The server collects the fee through the payment gateway and logs a collection completion message.

[1477] Step 2:

[1478] The server distributes the collected fees to guides, cultural facilities, and system operators.

[1479] Input: Collected fee data

[1480] Output: Distributed fees to each party

[1481] Specific operation: The server executes the distribution algorithm and sends a distribution completion message to each participant.

[1482] (Application example 1)

[1483] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1484] Conventional exhibit viewing systems at cultural facilities have not fully realized remote guided viewing, which has led to issues such as users being unable to effectively enjoy the exhibits using the latest technology. Furthermore, two-way communication, appropriate feedback collection, and matching of optimal guides based on user interests are also insufficient, hindering progress in the digitalization of cultural facilities.

[1485] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1486] In this invention, the server includes means for receiving exhibit data provided by the cultural facility and saving it as a digital archive, means for generating a three-dimensional virtual space based on the digital archive and appropriately arranging exhibits within the virtual space, means for inputting and managing guide profile information and specialty information, means for inputting and displaying user interests and desired exhibit content, means for matching the optimal guide with the user based on the input guide information and user information, means for the guide to guide the virtual space and provide real-time commentary, two-way communication means for users to submit questions and comments about the exhibits in real time via smartphones or tablet devices, means for collecting and analyzing feedback from users, means for providing the collected feedback to the guide and the cultural facility, and means for collecting tour fees from users and distributing the fees to the guide, the cultural facility, and the system operator. This enables the provision of high-quality cultural experiences remotely, matching users with the optimal guide based on their interests, providing real-time commentary and two-way communication, and analyzing and providing feedback.

[1487] A "cultural facility" is a place that displays exhibits and provides education and entertainment to the public, including art galleries, museums, and galleries.

[1488] "Exhibits" are objects such as artworks, historical materials, documents, etc. that are displayed in cultural institutions and that convey cultural or academic knowledge.

[1489] A "digital archive" is a system in which images, videos, and text data of exhibits are stored in digital format and managed as a database for future reference and search.

[1490] A "three-dimensional virtual space" is a virtual three-dimensional environment generated using computer technology, in which users can move freely within the space and visually appreciate exhibits and buildings that do not physically exist.

[1491] "Profile information" refers to data that includes information such as a guide's background, expertise, and track record, and this information is used to select and match guides.

[1492] "Specialty information" is information about a field or theme in which the guide is particularly knowledgeable, which enables the guide to provide specialized commentary and guidance to users.

[1493] "Matching" refers to the process of matching the most suitable guide with the user based on the user's interests and desired exhibits, as well as the guide's profile information and area of ​​expertise.

[1494] "Real-time commentary" means that a guide provides real-time audio and text explanations about exhibits and themes while guiding users through a virtual space, allowing users to obtain information instantly.

[1495] "Two-way communication" is a function that allows users to send questions and comments to guides and systems in real time via smartphones or tablet devices, engaging in two-way communication.

[1496] "Feedback" refers to the opinions and impressions that users enter after completing a tour, and this data is used to improve guides and cultural facilities.

[1497] "Fee collection" refers to the process of receiving tour fees from users through their payment method and distributing those fees to guides, cultural facilities, and system operators through the system.

[1498] A "generative AI model" is an artificial intelligence model that learns from large amounts of data to generate language and analyze data in a natural, human-like manner, and is used to analyze user feedback and select the optimal guide.

[1499] A "prompt" is an instruction given to a generative AI model to perform a specific task, allowing the AI ​​model to analyze and generate based on the specified content.

[1500] This invention is a system that enables users in remote locations to view exhibits at cultural facilities with a guide. The program for this system comprises the following steps:

[1501] The server receives data (images, videos, text) about exhibits provided by cultural facilities and stores it as a digital archive. Specifically, the server uses image analysis technology (such as OpenCV) to extract detailed information and stores it in a database (such as SQLite). This makes it possible to manage metadata (such as the name of the work, artist, era, and description) for each exhibit.

[1502] Next, the server generates a 3D virtual space based on the digital archive. Exhibits are placed in this virtual space in the same way as they would be in a real cultural facility. For example, the server uses a 3D rendering engine such as Unity or Unreal Engine to recreate a virtual art gallery or museum. The server also manages the placement data of the exhibits and places them in the appropriate locations.

[1503] Next, the guide enters and registers their profile information (career history, expertise, achievements) and areas of expertise into the server using a dedicated application. This saves and manages the guide information in a database. For example, if a guide enters their profile as a "historian," this information is saved in the system.

[1504] Users can create an account through the application and enter their interests and desired exhibits. They can also enter the time they are available. This information is sent to the server and analyzed. The server then matches users with the most suitable guide based on their interests. For example, if a user expresses an interest in "Renaissance paintings," the server will select a guide with that expertise.

[1505] At the designated date and time, the guide and user access the application. The guide will guide the user through the virtual space and provide real-time commentary. Users can send questions and comments about the exhibits in real time via their smartphones or tablets. This two-way communication function allows users to view the exhibits while communicating directly with the guide.

[1506] After the tour ends, feedback is collected from users. The server analyzes the collected feedback and provides it to guides and cultural facilities. The generative AI model also analyzes the feedback and suggests improvements to guide behavior and optimal placement of exhibits. Furthermore, by inputting prompts such as "Generate optimal guide suggestions based on the user's interests and feedback" into the generative AI model, the selection of the optimal guide can be automated.

[1507] Regarding fee collection, tour fees are collected from users and distributed to guides, cultural facilities, and system operators. For example, fees can be collected using credit cards or electronic payment services (such as PayPal), and the system automatically distributes them to each party.

[1508] As a specific example, if User A uses the app and inputs that he or she is interested in Renaissance paintings, the server will match the user with Guide B, who has specialized knowledge, based on that information. The user accesses the app at the specified date and time, and Guide B will guide the user through the virtual museum. After the tour ends, User A provides feedback, which is saved on the server. The generative AI model then analyzes the feedback and suggests areas for improving Guide B's explanation and optimal placement of exhibits.

[1509] Example prompt: "Generate optimal guide suggestions based on the user's interests and feedback."

[1510] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1511] Step 1: Receive exhibit data from cultural facilities and store it as a digital archive

[1512] The server receives images, videos, and text data of exhibits provided by cultural facilities via an API. This data is sent to the server, where image analysis technology (e.g., OpenCV) is used to extract detailed information (such as the name of the work, the artist, the year, and a description) and store it in a database (e.g., SQLite). The input is data about the exhibits, and the output is digitally archived data.

[1513] Step 2: Generate a 3D virtual space based on the digital archive

[1514] The server generates a virtual space using a 3D rendering engine such as Unity or Unreal Engine based on the stored digital archive. Exhibits are placed in appropriate positions within the virtual space. The input is digital archive data, and the output is a 3D virtual space.

[1515] Step 3: Enter and manage your guide's profile information and specialty information

[1516] The guide uses a terminal to input his / her profile information (career history, expertise, achievements) and areas of expertise. This information is sent to the server and stored in a database. The input is the guide's profile information, and the output is the guide information stored in the database.

[1517] Step 4: Enter your interests and desired exhibits

[1518] Users create an account using their device and enter their interests (the exhibits and themes they wish to view) and the time they have available. This information is sent to the server and analyzed. The input is the user's interests and desired exhibits, and the output is the analyzed user information.

[1519] Step 5: Matching the best guide with the user

[1520] The server performs appropriate matching based on the input guide information and user information. Using a generative AI model, it inputs a prompt such as "Generate optimal guide suggestions based on the user's interests and feedback" and selects the optimal guide. The input is guide information and user information, and the output is the matched guide and user.

[1521] Step 6: A guide will guide you through the virtual space

[1522] At the specified date and time, the guide and user access the application using their terminals. The guide will guide the user through the virtual space in real time and provide explanations of the exhibits. The input is the guide and user matched with the specified date and time, and the output is the real-time guided tour.

[1523] Step 7: Send your questions and comments via interactive communication

[1524] Visitors can send questions and comments about the exhibits in real time via smartphones or tablet devices, and the guide will respond accordingly. The input is the visitor's questions and comments, and the output is the guide's responses.

[1525] Step 8: Collect and analyze feedback

[1526] After the tour ends, feedback is collected from the user. The server analyzes the collected feedback and provides it to the guide and cultural facilities. It also analyzes the feedback using a generative AI model and proposes improvements to the guide's behavior and optimal placement of exhibits. The input is the user's feedback, and the output is the analysis results and proposals.

[1527] Step 9: Toll collection and distribution

[1528] The server collects tour fees from users and distributes them to guides, cultural facilities, and system operators using credit card payments or electronic payment services (such as PayPal). The input is the user's payment information, and the output is the distributed fee.

[1529] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1530] The system of the present invention allows remote users to view exhibits at cultural facilities with a guide, and also combines an emotion engine that recognizes the user's emotions. The system's program consists of many modules and methods, which provide users with a high-quality cultural experience and promote the improvement of guides and cultural facilities.

[1531] The main components of this system are as follows:

[1532] Digital archiving of art museums

[1533] server:

[1534] Images, videos, and text data of exhibits are received from cultural facilities via API. This data is stored on a server, and metadata (such as the name of the work, artist, year, and description) for each exhibit is analyzed and saved as a digital archive. For example, image data of a "famous painting" is received, and detailed information is extracted using image analysis technology and saved in the digital archive.

[1535] Creating a virtual museum

[1536] server:

[1537] The server generates a three-dimensional virtual space based on the stored digital archives. Within this virtual space, exhibits are arranged in the same way as they would be in a real cultural facility. For example, the interior of a "historical building" can be recreated, with "ancient artifacts" on display placed in appropriate locations.

[1538] Guide registration and management

[1539] Terminal (Guide):

[1540] Guides use a dedicated application to register their profile information (career history, expertise, achievements) and areas of expertise. This information is sent to the server and stored in a database. For example, a guide can enter a profile as a "historian," and that information is stored and managed within the system.

[1541] User registration and matching system

[1542] Terminal (user):

[1543] Users create an account through the application and enter their interests (the exhibits and themes they wish to view) and available time. The server matches them with the most suitable guide based on the information they enter. For example, if a user expresses an interest in "Renaissance paintings," the server will select a guide who is knowledgeable about Renaissance paintings.

[1544] Tour execution and feedback collection

[1545] Terminals (Guide and User):

[1546] At the specified date and time, the guide and user access the application. The guide guides the user through the virtual museum and provides real-time commentary. Users can share questions and comments through chat or voice communication. For example, while the guide explains "modern sculpture" to the user, the user can freely ask questions and provide feedback after the explanation.

[1547] server:

[1548] The server receives feedback from users and stores it in a database. The server analyzes the feedback and provides detailed reports to guides and cultural facilities.

[1549] Incorporating an emotion engine

[1550] Terminal (user):

[1551] The system uses an emotion engine that analyzes the user's facial expressions, tone of voice, etc. to monitor the user's emotional state in real time. For example, if the user feels surprise or excitement after seeing an exhibit, that emotional state will be analyzed.

[1552] server:

[1553] The emotion engine analyzes the user's emotional data and provides feedback to the guide based on that information, such as instructing the guide to provide a more detailed explanation of an exhibit that the user is interested in.

[1554] Revenue Share Distribution

[1555] server:

[1556] The server collects the tour fee from the user and distributes it to the guide, the cultural facility, and the system operator. For example, if the tour fee is paid by credit card, the server automatically distributes the fee to each party.

[1557] By integrating these functions, this system simultaneously digitizes cultural facilities, improves user convenience, and increases revenue. Furthermore, by incorporating an emotion engine, it can enrich the user experience and provide data useful for improving the operation of guides and cultural facilities. This is expected to promote financial support and digitalization for cultural facilities, and improve user cultural satisfaction.

[1558] The processing flow will be explained below.

[1559] Digital archiving of art museums

[1560] Step 1:

[1561] server:

[1562] Images, videos, and text data provided by cultural facilities are received via API.

[1563] Step 2:

[1564] server:

[1565] Store the received data in local storage or cloud storage.

[1566] Step 3:

[1567] server:

[1568] Metadata (title, artist, year, description) is extracted from the stored data using an image analysis algorithm.

[1569] Step 4:

[1570] server:

[1571] The extracted metadata is registered in a database.

[1572] Creating a virtual museum

[1573] Step 1:

[1574] server:

[1575] Using 3D modeling tools, a three-dimensional virtual space is generated based on the digital archive.

[1576] Step 2:

[1577] server:

[1578] The coordinate data of the exhibits is set within the generated virtual space.

[1579] Step 3:

[1580] server:

[1581] Based on the coordinate data, the exhibits are arranged and interactive functions are added.

[1582] Guide registration and management

[1583] Step 1:

[1584] Terminal (Guide):

[1585] The guide opens a dedicated application and enters their profile information and areas of expertise.

[1586] Step 2:

[1587] Terminal (Guide):

[1588] Add details about yourself, your achievements, etc.

[1589] Step 3:

[1590] server:

[1591] Receives guide input information and stores it in a database.

[1592] User registration and matching system

[1593] Step 1:

[1594] Terminal (user):

[1595] The user opens the dedicated application and creates an account.

[1596] Step 2:

[1597] Terminal (user):

[1598] Enter your interests, desired exhibits, and available time.

[1599] Step 3:

[1600] server:

[1601] It takes the user's input information and searches the database for the appropriate guide.

[1602] Step 4:

[1603] server:

[1604] Using an AI algorithm, it matches the user's interests, the guide's areas of expertise, and exhibit information.

[1605] Step 5:

[1606] server:

[1607] The matching results are presented to the user, and the user selects a guide.

[1608] Tour execution and feedback collection

[1609] Step 1:

[1610] Terminals (Guide and User):

[1611] The Guide and user access the application at the specified date and time.

[1612] Step 2:

[1613] Terminal (Guide):

[1614] A guide will guide you through the virtual museum and provide real-time commentary on the exhibits.

[1615] Step 3:

[1616] Terminal (user):

[1617] Users share questions and thoughts through chat functionality and voice communication.

[1618] Step 4:

[1619] Terminal (user):

[1620] After the tour, the user fills out a feedback form.

[1621] Step 5:

[1622] server:

[1623] Receive user feedback and store it in a database.

[1624] Step 6:

[1625] server:

[1626] The feedback data is analysed and provided to guides and cultural institutions as detailed reports.

[1627] Incorporating an emotion engine

[1628] Step 1:

[1629] Terminal (user):

[1630] Activate an emotion engine that analyzes the user's facial expressions and tone of voice.

[1631] Step 2:

[1632] Terminal (user):

[1633] The emotion engine analyzes the user's emotional state in real time and sends the data to the server.

[1634] Step 3:

[1635] server:

[1636] It receives emotional data sent in real time and generates feedback to the guide based on the analysis results.

[1637] Step 4:

[1638] Terminal (Guide):

[1639] The guide receives real-time feedback and adjusts the content and progress of the explanation based on the user's emotional state.

[1640] Revenue Share Distribution

[1641] Step 1:

[1642] server:

[1643] The tour fee is collected from the user via credit card or electronic payment.

[1644] Step 2:

[1645] server:

[1646] Temporarily store the collected fees.

[1647] Step 3:

[1648] server:

[1649] The fees will be distributed to guide, cultural institution and platform accounts based on a revenue share percentage.

[1650] Step 4:

[1651] server:

[1652] Process payments to each distribution recipient.

[1653] Through these processing steps, the system of the present invention simultaneously digitizes cultural facilities, improves user convenience, and increases revenue. Furthermore, by incorporating an emotion engine, it can enrich the user experience and provide data that is useful for improving the operation of guides and cultural facilities.

[1654] Example 2

[1655] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1656] When remote visitors view exhibits at cultural facilities, they need a high-quality, immersive experience while also being able to effectively collect, manage, and utilize their feedback on guides and cultural facilities. A system is also needed that can grasp the emotional state of visitors in real time and enable guides to respond appropriately based on that information. Fair and efficient distribution of revenue is also required.

[1657] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1658] In this invention, the server includes means for receiving exhibit data provided by the cultural facility and saving it as a digital archive, means for generating a three-dimensional virtual space based on the digital archive and appropriately arranging the exhibits within the virtual space, and means for inputting and managing guide profile information and specialty information. This allows remote users to enjoy a high-quality, immersive cultural experience while also enabling monitoring of the users' emotional states and real-time responses, and realizing fair and efficient distribution of revenue.

[1659] "Exhibits" refers to artworks, historical artifacts, artifacts, and other cultural assets that are made available to the public at cultural institutions.

[1660] A "guide" refers to a person who has knowledge of the exhibits at a cultural facility and provides explanations to visitors.

[1661] "User" refers to an individual in a remote location who wishes to view the exhibits at a cultural facility.

[1662] A "digital archive" refers to a system that stores and manages exhibit data (images, videos, text, etc.) in digital format.

[1663] "Three-dimensional virtual space" refers to a three-dimensional virtual space generated using digital technology that recreates the interior of an actual cultural facility.

[1664] "Profile information" refers to information about the guide, such as their background, expertise, and achievements.

[1665] "Emotional state" refers to emotions (surprise, excitement, joy, etc.) analyzed from the user's facial expressions and tone of voice.

[1666] "Feedback" refers to impressions, opinions, and ratings collected from users regarding exhibits and guides.

[1667] "Distribution" refers to the fair and efficient division of revenues (e.g., tour fees) among guides, cultural institutions, and system operators.

[1668] The system of the present invention allows remote users to view exhibits at cultural facilities with a guide, and also combines an emotion engine that recognizes the user's emotions. The system is composed of multiple modules and means, providing users with a high-quality cultural experience and promoting the improvement of guides and cultural facilities.

[1669] Digital archiving of art museums

[1670] server:

[1671] Images, videos, and text data of exhibits are received from cultural facilities via API. For example, TensorFlow is used as a specific image analysis technology to extract the name of the work and information about the artist from the received image data. This data is stored on a server as metadata (work name, artist, year, description, etc.) and managed as a digital archive. As a specific example, image data of a famous painting is received, and the name of the work and information about the artist are extracted using image analysis technology.

[1672] Creating a virtual museum

[1673] server:

[1674] A 3D virtual space is generated based on the stored digital archives. 3D models of the exhibits are created using 3D engines such as Unity or Unreal Engine, and then placed inside the virtual museum. Specifically, the interior of a "historical building" is recreated, and the "ancient artifacts" to be displayed there are placed in the appropriate locations.

[1675] Guide registration and management

[1676] Terminal (navigator):

[1677] Guides use a dedicated application to register their profile information (career history, expertise, achievements) and areas of expertise. This information is sent to the server and stored in a database. For example, if a guide enters their profile as a "historian," the information is sent to the server in real time and stored in the database.

[1678] User registration and matching system

[1679] Terminal (user):

[1680] Users create an account through the application and input their interests (the exhibits and themes they wish to view) and available time. The server matches them with the most suitable guide based on the information they input. For example, if a user expresses an interest in "Renaissance paintings," the server will select a guide who is knowledgeable about Renaissance paintings.

[1681] Tour execution and feedback collection

[1682] Terminals (instructor and user):

[1683] At the designated date and time, the guide and user access the application. The guide will guide the user through the virtual museum and provide real-time commentary. Users can share their questions and impressions through chat and voice communication. For example, while the guide is explaining about "modern sculpture," users can freely ask questions and provide feedback after the explanation.

[1684] server:

[1685] Feedback from users is received and stored in a database. The server analyzes the feedback and provides detailed reports to guides and cultural facilities.

[1686] Incorporating an emotion engine

[1687] Terminal (user):

[1688] The system uses an emotion engine that analyzes the user's facial expressions and tone of voice to monitor the user's emotional state in real time. For example, the system uses the emotion recognition API from Microsoft Azure Cognitive Services as a specific analysis technology. For example, if a user feels surprise or excitement when looking at an exhibit, that emotional state is analyzed.

[1689] server:

[1690] The emotion engine analyzes the user's emotional data and provides feedback to the guide based on that information. For example, it can instruct the guide to provide a more detailed explanation of an exhibit that the user is interested in.

[1691] Revenue Share Distribution

[1692] server:

[1693] The server collects the tour fee from the user and distributes it to the guide, the cultural facility, and the system operator. Specifically, it uses a payment gateway API such as Stripe for payment processing. When the tour fee is paid by credit card, the server automatically distributes the fee to each party.

[1694] Specific examples and examples of prompts for generative AI models

[1695] As a concrete example, consider a case where a user wants to view a "famous Renaissance painting." In this case, the server selects a "guide with expertise in Renaissance painting." If the emotion engine detects the user's excitement, it instructs the guide to provide more detailed information about the exhibit. An example of a prompt sentence for the generative AI model is as follows:

[1696] 1. "User is interested in Renaissance paintings. Please select relevant guides."

[1697] 2. "The user seems surprised by the exhibit. Please instruct the guide to provide a detailed explanation."

[1698] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1699] Step 1:

[1700] The server receives and stores data from the cultural institution.

[1701] Input: Images, videos, and text data of exhibits provided by cultural facilities via API

[1702] Specific operation: The server receives data on the exhibits using an API. To do this, it uses image analysis technology such as TensorFlow to extract the name of the work and information about the artist from the received image data and saves it as metadata.

[1703] Output: The analyzed metadata and digital data of the exhibits are saved on the server.

[1704] Step 2:

[1705] The server generates a three-dimensional virtual space.

[1706] Input: Stored digital archive data

[1707] How it works: The server uses a 3D engine such as Unity or Unreal Engine to generate a 3D virtual space based on the stored digital archive data. 3D models of the exhibits are generated and placed appropriately within the virtual museum.

[1708] Output: A 3D virtual museum space accessible to visitors and guides

[1709] Step 3:

[1710] The device (guide) registers and manages profile information

[1711] Input: Guide's profile information (career history, expertise, achievements) and area of ​​expertise

[1712] Specific operation: The guide uses a dedicated application to enter his / her profile information and send it to the server. This information is saved in real time on the server and managed in a database.

[1713] Output: Guide profile information stored on the server

[1714] Step 4:

[1715] The device (user) inputs interest information

[1716] Input: User's interests (exhibits and themes they wish to see) and available time input through the application

[1717] How it works: A user uses the application to input the exhibits and themes they wish to see, as well as the time they have available. This information is sent to the server and stored in a database.

[1718] Output: User interest information and available time stored on the server

[1719] Step 5:

[1720] The server matches guides with users.

[1721] Input: Guide profile information and user interest information stored in the database

[1722] How it works: The server collates the user's interests and automatically selects the most suitable guide. This process may involve the use of generative AI models. For example, a prompt might read, "The user is interested in Renaissance paintings. Please select a relevant guide."

[1723] Output: A properly matched guide-user pair

[1724] Step 6:

[1725] Conducting guided tours and providing real-time commentary

[1726] Input: Pre-matched guide and user, 3D virtual space data

[1727] Specific operation: At the specified date and time, the guide and user access the application and begin a tour of the virtual museum. The guide explains the exhibits in real time, and users share their questions and impressions using the chat function and voice communication.

[1728] Output: Real-time commentary and user questions and comments

[1729] Step 7:

[1730] The server collects and analyzes the feedback

[1731] Input: User feedback (chat messages, impressions, ratings)

[1732] Specific operation: After the tour ends, the server collects feedback from users and stores it in a database. It then analyzes the feedback and provides a detailed report to the guide and the cultural facility.

[1733] Output: Parsed feedback report

[1734] Step 8:

[1735] Monitor emotional states using the Emotion Engine

[1736] Input: Real-time user facial expressions and tone of voice

[1737] Specific behavior: Using an emotion engine (e.g., Microsoft Azure Cognitive Services), the user's facial expressions and tone of voice are analyzed in real time to monitor their emotional state. The analysis results are sent to the server. An example prompt might be, "The user seems surprised by the exhibits. Please instruct the guide to provide a detailed explanation."

[1738] Output: Real-time analyzed emotion data

[1739] Step 9:

[1740] The server provides feedback based on emotion data

[1741] Input: Parsed emotion data

[1742] Specific operation: The server provides real-time feedback to the guide based on the analyzed emotion data, allowing the guide to provide more detailed information about the exhibits that the user is interested in.

[1743] Output: Feedback instructions based on emotion data

[1744] Step 10:

[1745] The server collects and distributes the tour fee.

[1746] Input: Tour payment information from user

[1747] Specific operation: The server uses a payment gateway API such as Stripe to collect tour fees from users, and distributes the collected fees fairly and efficiently among guides, cultural facilities, and system operators.

[1748] Output: Tour fare distributed to each party

[1749] (Application example 2)

[1750] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1751] Conventional systems for remote viewing of exhibits at cultural facilities have had the problem of being unable to provide a personalized experience based on the user's emotions and interests. In addition, they lacked the functionality for users to obtain information interactively about the exhibits, which could result in a decrease in the user's understanding of the explanations and a decrease in their satisfaction.

[1752] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for receiving exhibit data provided by a cultural facility and saving it as a digital archive; means for generating a three-dimensional virtual space based on the digital archive and appropriately arranging exhibits in the virtual space; means for inputting and managing guide profile information and specialty information; means for inputting and displaying the user's interests and desired exhibit content; means for matching the optimal guide with the user based on the input guide information and user information; means for the guide to guide the virtual space and provide commentary in real time; means for collecting and analyzing feedback from the user; means for providing the collected feedback to the guide and the cultural facility; means for collecting tour fees from the user and distributing the fees to the guide, the cultural facility, and the system operator; and means for analyzing the user's emotions and adjusting the guide's commentary based on the analysis data. This allows users to obtain information about the exhibits interactively and provides a personalized experience tailored to the user's emotions and interests.

[1753] "Cultural facilities" are facilities that collect, preserve, and display cultural exhibits and materials, such as art galleries, museums, and libraries.

[1754] An "exhibit" is a tangible item, such as a work of art, historical artifact, scientific specimen, or document, that is displayed to visitors at a cultural institution.

[1755] A "digital archive" is an electronic database for storing and managing images, videos, text data, etc. of exhibits in digital format.

[1756] A "three-dimensional virtual space" is a three-dimensional virtual space generated by a computer, and is a virtual environment that a user can experience using a virtual reality device or the like.

[1757] A "guide" is a person or system that has specialized knowledge about cultural facilities and exhibits and provides explanations and guidance to users.

[1758] "Profile Information" refers to personal information about a guide, such as their background, expertise, achievements, and areas of expertise.

[1759] "Interest" refers to the degree of interest a visitor has in exhibits or themes that they find particularly appealing.

[1760] "Matching" is the process of selecting the most suitable guide based on the user's interests and desired exhibits, and connecting the user with the guide.

[1761] "Feedback" refers to reactions such as impressions, evaluations, and opinions from users, and is the process of collecting and analyzing these to help improve services.

[1762] "Emotion analysis" is a technology that analyzes a user's facial expressions and tone of voice to assess their emotional state in real time.

[1763] "Personalization" means individually tailoring and providing services and content to suit each user's interests and emotions.

[1764] An embodiment of the present invention provides a remote guided viewing experience of exhibits at a cultural facility by linking a server, terminals (guides, users), and an emotion analysis system. Specific embodiments are described below.

[1765] 1. Digital Archiving

[1766] The server receives images, videos, and text data of exhibits provided by cultural facilities via an API. This data is stored in a database on the server, and metadata (such as the title of the work, artist, year, and description) for each exhibit is analyzed and saved as a digital archive. Specifically, a cloud server on AWS (Amazon Web Services) is used, and the saved data is managed in an SQL database.

[1767] 2. Creating a virtual store

[1768] The server generates a 3D virtual space based on the stored digital archives. Unity and Unreal Engine are used to appropriately position exhibits within this virtual space. This virtually recreates the layout of the actual cultural facility, providing users with a realistic experience.

[1769] 3. Guide registration and management

[1770] Guides use a dedicated application to enter their profile information (career history, expertise, achievements) and areas of expertise, and then send this information to a server. This data is stored in a database on the server and later used to match users. The device can be a smartphone, tablet, or PC.

[1771] 4. User Registration and Matching System

[1772] Users create an account through the application and enter their interests (exhibits and themes they wish to see) and available times. The server analyzes this information and matches them with the most suitable guide. In particular, frameworks such as React and AngularJS are used for the user interface.

[1773] 5. Product Tour and Feedback Collection

[1774] At a specified date and time, the guide and user access the application. The guide navigates the virtual space and provides commentary in real time. Users share questions and impressions through chat and voice communication. Commentary is provided using a combination of speech recognition and text conversion systems, and feedback is sent to a server for storage. Python and TensorFlow are used for feedback analysis.

[1775] 6. Incorporating an Emotion Analysis System

[1776] An emotion engine that analyzes the user's facial expressions and tone of voice is installed on the device to monitor the user's emotional state in real time. This is done using OpenCV and Azure Emotion API. Emotion data is sent to a server, which provides feedback to the guide based on the analysis results.

[1777] 7. Revenue Share Distribution

[1778] The server collects the tour fee from the user and automatically distributes it to the guide, the cultural facility, and the system operator. Payment is processed using electronic payment services such as Stripe and PayPal.

[1779] Specific examples

[1780] For example, if a user expresses interest in a "historical building" exhibit in a virtual store, the system will use that information to match with a guide knowledgeable in history and conduct a tour. If the user shows excitement at an "ancient relic" displayed during the tour, the emotion engine will analyze this emotion and prompt the guide to provide a detailed explanation.

[1781] Example prompts for generative AI models

[1782] "Analyze how users are responding to this product. Based on image and audio data, determine whether their reactions are positive or negative based on facial expressions and tone of voice, and reflect this in a detailed guide."

[1783] This allows visitors to interact with the exhibits and obtain information about them, and also enables a personalized experience based on the visitors' emotions and interests.

[1784] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1785] Step 1:

[1786] The server receives images, videos, and text data of exhibits provided by cultural facilities via API. The input data is the exhibit's metadata (work title, artist, year, description, etc.), which is stored in a database. The server analyzes the stored data and organizes it into a digital archive. Specifically, it performs an INSERT operation on the database table, saves the data, and generates a CONFIRM message.

[1787] Step 2:

[1788] The server generates a three-dimensional virtual space using Unity or Unreal Engine based on the saved digital archive. The input data is the digital archive data saved in step 1. This also includes location and layout information for the exhibits to be placed appropriately within the virtual space. Specifically, the server uses Unity's Scene Builder function to place the 3D models of the exhibits within the scene and generate EXPORTED scene data.

[1789] Step 3:

[1790] The terminal (guide) uses a dedicated application to input their own profile information (career history, expertise, achievements) and areas of expertise. The input data is the guide's personal information and expertise information, and is sent to the server. The server stores the sent data in a database. Specifically, the terminal receives the input data through the GUI, sends a POST request to the server, and the server inserts it into the database.

[1791] Step 4:

[1792] The device (user) creates an account through the application and inputs their interests (exhibits and themes they wish to view) and available times. The input data is the user's personal information and interest data, and is sent to the server. The server analyzes this and matches them with the most suitable guide. Specifically, the server applies guide information and a matching algorithm based on the user's interest data to generate MATCHED parameters.

[1793] Step 5:

[1794] At the specified date and time, the guide and user access the application. The guide navigates the virtual space and provides commentary in real time. The input data is the scene data of the virtual space and the commentary, which is displayed and played on the device. Specifically, the device renders the virtual space data and displays the guide's commentary in audio or text.

[1795] Step 6:

[1796] An emotion engine that analyzes the user's facial expressions and tone of voice is installed on the device, monitoring the user's emotional state in real time. The input data is images of the user's facial expressions and voice data, which is analyzed to generate emotional state data. Specifically, the device acquires data from the camera and microphone, runs an emotion analysis algorithm (OpenCV or Azure Emotion API), and sends the analysis results to the server.

[1797] Step 7:

[1798] The server provides feedback to the guide based on the emotion analysis results sent by the user. The input data is emotional state data, and based on this, feedback data is generated to adjust the guide's explanations. Specifically, the server analyzes the emotion data, generates a feedback message instructing the necessary action (strengthening the explanation, providing more detailed explanations, etc.), and sends a push notification to the guide's device.

[1799] Step 8:

[1800] After the tour ends, the server collects feedback from users and stores it in a database. The input data is the user's feedback information, which is analyzed to generate report data to be provided to cultural facilities and guides. Specifically, the server receives the feedback information, applies an analytical algorithm (such as natural language processing) to create a report, and sends it to the relevant parties.

[1801] Step 9:

[1802] The server collects tour fees from users and automatically distributes them to guides, cultural facilities, and system operators. The input data is payment information and distribution ratio data, and distribution transactions are carried out based on this. Specifically, the server collects fees through a payment gateway (Stripe or PayPal) and automatically makes payments to the parties involved based on the pre-set distribution ratio.

[1803] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1804] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1805] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1806] [Fourth embodiment]

[1807] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1808] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1809] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1810] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1811] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1812] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1813] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1814] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1815] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1816] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1817] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1818] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1819] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1820] The system of the present invention allows users to view exhibits at cultural facilities remotely and provides guided tours. The system's program consists of numerous modules and tools, providing users with a high-quality cultural experience.

[1821] The main components of this system are as follows:

[1822] Digital archiving of art museums

[1823] server:

[1824] Images, videos, and text data of exhibits are received from cultural facilities via API. This data is stored on the server and analyzed to register metadata (title, artist, era, description, etc.) for each exhibit. As a specific example, the server receives image data of a "famous painting," extracts detailed information about it using image analysis technology, and stores it in a digital archive.

[1825] Creating a virtual museum

[1826] server:

[1827] The server generates a three-dimensional virtual space based on the stored digital archives. Within this virtual space, exhibits are arranged in the same way as they would be in a real cultural facility. For example, the server recreates the interior of a "historical building" and places the "ancient artifacts" on display in the appropriate locations.

[1828] Guide registration and management

[1829] Terminal (Guide):

[1830] Guides use a dedicated application to register their profile information (career history, expertise, achievements) and areas of expertise. This information is sent to the server and stored in a database. For example, a guide can enter a profile as a "historian," and that information is stored and managed within the system.

[1831] User registration and matching system

[1832] Terminal (user):

[1833] Users create an account through the application and enter their interests (the exhibits and themes they wish to view) and available time. The server matches them with the most suitable guide based on the information they enter. For example, if a user expresses an interest in "Renaissance paintings," the server will select a guide who is knowledgeable about Renaissance paintings.

[1834] Tour execution and feedback collection

[1835] Terminals (Guide and User):

[1836] At the specified date and time, the guide and user access the application. The guide guides the user through the virtual museum and provides commentary in real time. Users can share questions and impressions through chat or voice communication. After the tour ends, the user fills out a feedback form and sends it to the server. The server collects and analyzes the feedback and provides it to the guide and cultural facility. For example, while the guide explains to the user about "modern sculpture," the user is free to ask questions and provide feedback after the tour ends.

[1837] Revenue Share Distribution

[1838] server:

[1839] The server collects the tour fee from the user and distributes it to the guide, the cultural facility, and the system operator. For example, if the tour fee is paid by credit card, the server automatically distributes the fee to each party.

[1840] By integrating these functions, this system will solve the problems of cultural facilities, such as a lack of funding, delays in digitalization, and difficulty in accessing the system for users, and improve the quality of the cultural experience. Each function has been explained with concrete examples, and it is expected that this will increase the revenue of cultural facilities and improve user satisfaction.

[1841] The processing flow will be explained below.

[1842] Digital archiving of art museums

[1843] Step 1:

[1844] server:

[1845] Images, videos, and text data provided by cultural facilities are received via API.

[1846] Step 2:

[1847] server:

[1848] Store the received data in local storage or cloud storage.

[1849] Step 3:

[1850] server:

[1851] Metadata (title, artist, year, description) is extracted from the stored data using an image analysis algorithm.

[1852] Step 4:

[1853] server:

[1854] The extracted metadata is registered in a database.

[1855] Creating a virtual museum

[1856] Step 1:

[1857] server:

[1858] Using 3D modeling tools, a three-dimensional virtual space is generated based on the digital archive.

[1859] Step 2:

[1860] server:

[1861] The coordinate data of the exhibits is set within the generated virtual space.

[1862] Step 3:

[1863] server:

[1864] Based on the coordinate data, the exhibits are arranged and interactive functions are added.

[1865] Guide registration and management

[1866] Step 1:

[1867] Terminal (Guide):

[1868] The guide opens a dedicated application and enters their profile information and areas of expertise.

[1869] Step 2:

[1870] Terminal (Guide):

[1871] Also add detailed information such as your self-introduction and achievements.

[1872] Step 3:

[1873] server:

[1874] Receives guide input information and stores it in a database.

[1875] User registration and matching system

[1876] Step 1:

[1877] Terminal (user):

[1878] The user opens the dedicated application and creates an account.

[1879] Step 2:

[1880] Terminal (user):

[1881] Enter your interests, desired exhibits, and available time.

[1882] Step 3:

[1883] server:

[1884] It takes the user's input information and searches the database for the appropriate guide.

[1885] Step 4:

[1886] server:

[1887] Using an AI algorithm, it matches the user's interests, the guide's areas of expertise, and exhibit information.

[1888] Step 5:

[1889] server:

[1890] The matching results are presented to the user, and the user selects a guide.

[1891] Tour execution and feedback collection

[1892] Step 1:

[1893] Terminals (Guide and User):

[1894] The Guide and user access the application at the specified date and time.

[1895] Step 2:

[1896] Terminal (Guide):

[1897] A guide will guide you through the virtual museum and provide real-time commentary on the exhibits.

[1898] Step 3:

[1899] Terminal (user):

[1900] Users share questions and thoughts through chat functionality and voice communication.

[1901] Step 4:

[1902] Terminal (user):

[1903] After the tour, the user fills out a feedback form.

[1904] Step 5:

[1905] server:

[1906] Receive user feedback and store it in a database.

[1907] Step 6:

[1908] server:

[1909] The feedback data is analysed and provided to guides and cultural institutions as detailed reports.

[1910] Revenue Share Distribution

[1911] Step 1:

[1912] server:

[1913] The tour fee is collected from the user via credit card or electronic payment.

[1914] Step 2:

[1915] server:

[1916] Temporarily store the collected fees.

[1917] Step 3:

[1918] server:

[1919] The fees will be distributed to guide, cultural institution and platform accounts based on a revenue share percentage.

[1920] Step 4:

[1921] server:

[1922] Process payments to each distribution recipient.

[1923] Through these processing steps, the system of the present invention simultaneously realizes the digitization of cultural facilities, improved convenience for users, and increased profits.

[1924] Example 1

[1925] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1926] There is a need to provide the experience of viewing exhibits at cultural facilities to users in remote locations. Another challenge is to enable users to not only receive real-time commentary from guides, but also to select guides that suit their interests. Furthermore, it is necessary to collect and analyze user feedback and provide it to guides and cultural facilities to improve the quality of services. Finally, to ensure the economic sustainability of this system, an appropriate distribution of revenue shares must be addressed.

[1927] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1928] In this invention, the server includes: means for receiving exhibit data provided by the cultural facility and saving it as a digital archive; means for generating a three-dimensional virtual space based on the digital archive and appropriately arranging exhibits within the virtual space; means for inputting and managing guide profile information and specialty information; means for inputting and displaying user interests and desired exhibits; means for matching the optimal guide with the user based on the input guide information and user information; means for the guide and user to access the application at a specified date and time and provide real-time commentary; interactive means for users to share questions and impressions via chat or voice communication; means for collecting and analyzing feedback from users; means for providing the collected feedback to the guide and the cultural facility; and means for collecting tour fees from users and distributing the fees to the guide, the cultural facility, and the system operator. This allows users to view exhibits at the cultural facility in real time, even from remote locations, and enjoy an interactive experience tailored to their interests while listening to the guide's commentary. Furthermore, collecting and analyzing feedback and appropriately distributing fees improves the quality of the system and ensures its economic sustainability.

[1929] A "remotely located user" is a user who is physically located away from the cultural facility but who views the exhibits at the cultural facility through this system.

[1930] "Cultural facilities" refer to facilities that store and display exhibits, such as art galleries and museums.

[1931] "Exhibits" are objects or materials that are available for viewing, such as works of art, archaeological artifacts, or natural history exhibits, that are displayed at cultural institutions.

[1932] A "digital archive" is a database for digitally storing and managing data, including images, videos, and text of exhibits provided by cultural facilities.

[1933] A "three-dimensional virtual space" is a three-dimensional digital space generated by a computer, a virtual exhibition space that users can access and experience via the Internet.

[1934] A "guide" is a person with specialized knowledge who provides explanations and commentary on exhibits to visitors and leads tours.

[1935] "Profile information" refers to information about the guide's qualifications and abilities, such as the guide's background, expertise, and achievements.

[1936] "Specialty information" refers to information about areas or topics in which the guide is particularly knowledgeable.

[1937] "Interests" refer to the exhibits or themes that visitors are particularly interested in viewing.

[1938] "Desired exhibit content" refers to the specific exhibits or themes that the user would like to see on the tour.

[1939] "Matching" refers to the process of matching the most suitable guide with the user based on the guide information and user information entered.

[1940] "Application" refers to the software that allows guides and users to access the system and perform various operations.

[1941] "Interactive function" refers to a function that allows two-way communication, such as allowing users to ask questions about exhibits and obtain information in real time.

[1942] "Feedback" refers to impressions, evaluations, suggestions for improvement, and other information provided by users after the tour has ended.

[1943] "Feedback collection" refers to the process of collecting feedback provided by users as data.

[1944] "Feedback analysis" refers to the process of analyzing collected feedback to help improve the quality of services.

[1945] "Tour Fee" means the fee paid by a User to participate in a Guided Tour.

[1946] "Revenue share" is a system in which collected tour fees are distributed to guides, cultural facilities, and system operators.

[1947] MODE FOR CARRYING OUT THE INVENTION

[1948] The system of the present invention allows users in remote locations to view exhibits at cultural facilities and experience guided tours. This system operates in cooperation with a server, terminals, and users. A specific implementation method is described below.

[1949] Digital archiving of art museums

[1950] The server first receives data (images, videos, text, etc.) about exhibits from cultural facilities via an API. The received data is analyzed using image analysis software such as TensorFlow, and metadata such as the work's name, artist, era, and description is extracted. This data is then stored as a digital archive. For example, the server could receive image data of a "famous painting," extract its detailed information, and store it in a digital archive.

[1951] Creating a virtual museum

[1952] The server uses the stored digital archives to create a virtual space using 3D virtual space generation software such as Unity. Exhibits are placed in this virtual space just as they would be in a real cultural facility. For example, a historical building could be recreated, with ancient artifacts appropriately placed inside.

[1953] Guide registration and management

[1954] Guides use a dedicated terminal application to enter their profile information (career history, expertise, achievements) and areas of expertise. This information is sent to the server and stored in a database. For example, a guide enters their profile as a historian, and that information is stored in the system.

[1955] User registration and matching system

[1956] Users create an account through the terminal application and enter their interests, desired exhibits, and available times. Based on this information, the server selects the most suitable guide and matches them with the user. For example, if a user expresses an interest in Renaissance paintings, the server will select a guide who is an expert in that field.

[1957] Tour execution and feedback collection

[1958] At the specified date and time, the guide and user access the application. The guide guides the user through the virtual museum and provides real-time commentary using communication software such as Zoom or Microsoft Teams. Users can share questions and impressions through chat functions or voice communication. After the tour, the user fills out a feedback form and sends it to the server. The server collects and analyzes this feedback and provides it to the guide and cultural facility. For example, while the guide explains about modern sculpture to the user, the user can ask questions freely and provide their impressions after the tour.

[1959] Revenue Share Distribution

[1960] The server collects the tour fee from the user and distributes it to the guide, the cultural facility, and the system operator. For example, the server automatically distributes the tour fee paid by credit card to each party.

[1961] ---

[1962] Example prompt sentence:

[1963] 1. "I'm interested in Japanese history. I'd like to see exhibits from the Kamakura period. Which guide would be best?"

[1964] 2. "I'd like to tour a virtual art museum next Saturday. Please select a guide who specializes in 17th-century European painting."

[1965] 3. "I'd like a detailed explanation of the Mona Lisa. Could you please check the digital archive of this exhibit and explain it to me?"

[1966] This system allows users to view the rich exhibits at cultural facilities from remote locations and receive real-time commentary from expert guides. It also enables two-way communication, allowing for feedback to be used to improve the quality of the service. Furthermore, the system's sustainability is ensured through appropriate distribution of fees.

[1967] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1968] Program processing flow and steps

[1969] Digital archiving of art museums

[1970] Step 1:

[1971] The server receives images, videos, and text data of exhibits from cultural facilities via an API.

[1972] Input: Exhibit data from cultural facilities (images, videos, text)

[1973] Output: The received data is loaded into the server's memory.

[1974] Specific operation: The server calls the API and receives and displays exhibit data in real time.

[1975] Step 2:

[1976] The server analyzes the received data and extracts metadata using image analysis techniques such as TensorFlow.

[1977] Input: Received exhibit data

[1978] Output: Metadata such as title, artist, era, description, etc.

[1979] Specific operation: The server executes the image analysis model and displays the analysis progress on the monitoring screen.

[1980] Step 3:

[1981] The server stores the analyzed metadata in a digital archive.

[1982] Input: Extracted metadata

[1983] Output: Metadata stored in a digital archive database

[1984] Specific behavior: The server performs the save operation and logs a save completion message.

[1985] Creating a virtual museum

[1986] Step 1:

[1987] The server retrieves the necessary data from the stored digital archive.

[1988] Input: Digital Archive Database

[1989] Output: Loaded exhibit data

[1990] What happens: The server executes the database query and loads the retrieved data into memory.

[1991] Step 2:

[1992] The server generates a three-dimensional virtual space using Unity or similar software.

[1993] Input: Loaded exhibit data

[1994] Output: Generated 3D virtual space

[1995] Specific operation: The server starts the Unity engine and displays the progress of the virtual space generation in real time.

[1996] Step 3:

[1997] The server appropriately arranges the exhibits in the virtual space.

[1998] Input: Exhibit data, 3D virtual space

[1999] Output: The completed virtual museum

[2000] Specific operation: The server executes the data placement algorithm and logs a placement completion message.

[2001] Guide registration and management

[2002] Step 1:

[2003] The terminal (guide) uses a dedicated application to input profile information and area of ​​expertise information.

[2004] Input: Guide profile information, specialty information

[2005] Output: The information entered is displayed on the terminal.

[2006] What happens: The application displays a confirmation message and checks the validity of the input data.

[2007] Step 2:

[2008] The terminal (guide) sends the entered information to the server.

[2009] Input: Profile information, area of ​​expertise

[2010] Output: Guide information uploaded to the server

[2011] Specific behavior: The application displays the progress of data transmission and notifies you when transmission is complete.

[2012] Step 3:

[2013] The server stores the received guide information in a database.

[2014] Input: Guide profile information, specialty information

[2015] Output: Guide information stored in a database

[2016] Specific behavior: The server performs the save operation and logs a save completion message.

[2017] User registration and matching system

[2018] Step 1:

[2019] The terminal (user) creates an account through the application.

[2020] Input: User account information

[2021] Output: A confirmation message is displayed to create an account.

[2022] What happens: The application checks the validity of the information entered and displays a confirmation message.

[2023] Step 2:

[2024] The terminal (user) inputs their interests and desired exhibit contents.

[2025] Input: User's interests and desired exhibits

[2026] Output: The information entered is displayed on the terminal.

[2027] What happens: The application displays a confirmation message and checks the validity of the input data.

[2028] Step 3:

[2029] The terminal (user) sends input information to the server.

[2030] Input: User's interests and desired exhibits

[2031] Output: User information uploaded to the server

[2032] Specific behavior: The application displays the progress of data transmission and notifies you when transmission is complete.

[2033] Step 4:

[2034] The server selects the most suitable guide based on the input guide information and user information and matches them with the user.

[2035] Input: Guide information, user information

[2036] Output: The matched guide-user pair

[2037] What happens: The server runs the matching algorithm and logs a match completion message.

[2038] Tour execution and feedback collection

[2039] Step 1:

[2040] At the specified date and time, the terminals (guide and user) access the application.

[2041] Input: User and guide access information

[2042] Output: Successfully logged into the application

[2043] Specific behavior: The application performs login authentication and displays a login completion message.

[2044] Step 2:

[2045] The device (guide) will guide you through the virtual museum and provide real-time commentary using Zoom, Microsoft Teams, etc.

[2046] Input: Guide explanation, user access information

[2047] Output: Real-time commentary, video and audio data

[2048] Specific operation: The guide provides commentary and broadcasts video and audio data in real time.

[2049] Step 3:

[2050] Devices (users) share questions and impressions through chat functions and voice communication.

[2051] Input: Questions and comments from users

[2052] Output: Questions and comments for the guide

[2053] What it does: The application forwards your questions and comments to the guide in real time.

[2054] Step 4:

[2055] After the tour ends, the terminal (user) fills in a feedback form and sends it to the server.

[2056] Input: Feedback information

[2057] Output: Feedback sent to the server

[2058] What happens: The application displays the progress of the feedback submission and notifies you when it is complete.

[2059] Step 5:

[2060] The server collects and analyzes the feedback.

[2061] Input: Feedback data

[2062] Output: Parsed feedback information

[2063] What happens: The server runs the analysis algorithm and logs the results.

[2064] Revenue Share Distribution

[2065] Step 1:

[2066] The server collects the tour fee from the user.

[2067] Input: User's payment information

[2068] Output: Fees collected

[2069] Specific behavior: The server collects the fee through the payment gateway and logs a collection completion message.

[2070] Step 2:

[2071] The server distributes the collected fees to guides, cultural facilities, and system operators.

[2072] Input: Collected fee data

[2073] Output: Distributed fees to each party

[2074] Specific operation: The server executes the distribution algorithm and sends a distribution completion message to each participant.

[2075] (Application example 1)

[2076] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2077] Conventional exhibit viewing systems at cultural facilities have not fully realized remote guided viewing, which has led to issues such as users being unable to effectively enjoy the exhibits using the latest technology. Furthermore, two-way communication, appropriate feedback collection, and matching of optimal guides based on user interests are also insufficient, hindering progress in the digitalization of cultural facilities.

[2078] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2079] In this invention, the server includes means for receiving exhibit data provided by the cultural facility and saving it as a digital archive, means for generating a three-dimensional virtual space based on the digital archive and appropriately arranging exhibits within the virtual space, means for inputting and managing guide profile information and specialty information, means for inputting and displaying user interests and desired exhibit content, means for matching the optimal guide with the user based on the input guide information and user information, means for the guide to guide the virtual space and provide real-time commentary, two-way communication means for users to submit questions and comments about the exhibits in real time via smartphones or tablet devices, means for collecting and analyzing feedback from users, means for providing the collected feedback to the guide and the cultural facility, and means for collecting tour fees from users and distributing the fees to the guide, the cultural facility, and the system operator. This enables the provision of high-quality cultural experiences remotely, matching users with the optimal guide based on their interests, providing real-time commentary and two-way communication, and analyzing and providing feedback.

[2080] A "cultural facility" is a place that displays exhibits and provides education and entertainment to the public, including art galleries, museums, and galleries.

[2081] "Exhibits" are objects such as artworks, historical materials, documents, etc. that are displayed in cultural institutions and that convey cultural or academic knowledge.

[2082] A "digital archive" is a system in which images, videos, and text data of exhibits are stored in digital format and managed as a database for future reference and search.

[2083] A "three-dimensional virtual space" is a virtual three-dimensional environment generated using computer technology, in which users can move freely within the space and visually appreciate exhibits and buildings that do not physically exist.

[2084] "Profile information" refers to data that includes information such as a guide's background, expertise, and track record, and this information is used to select and match guides.

[2085] "Specialty information" is information about a field or theme in which the guide is particularly knowledgeable, which enables the guide to provide specialized commentary and guidance to users.

[2086] "Matching" refers to the process of matching the most suitable guide with the user based on the user's interests and desired exhibits, as well as the guide's profile information and area of ​​expertise.

[2087] "Real-time commentary" means that a guide provides real-time audio and text explanations about exhibits and themes while guiding users through a virtual space, allowing users to obtain information instantly.

[2088] "Two-way communication" is a function that allows users to send questions and comments to guides and systems in real time via smartphones or tablet devices, engaging in two-way communication.

[2089] "Feedback" refers to the opinions and impressions that users enter after completing a tour, and this data is used to improve guides and cultural facilities.

[2090] "Fee collection" refers to the process of receiving tour fees from users through their payment method and distributing those fees to guides, cultural facilities, and system operators through the system.

[2091] A "generative AI model" is an artificial intelligence model that learns from large amounts of data to generate language and analyze data in a natural, human-like manner, and is used to analyze user feedback and select the optimal guide.

[2092] A "prompt" is an instruction given to a generative AI model to perform a specific task, allowing the AI ​​model to analyze and generate based on the specified content.

[2093] This invention is a system that enables users in remote locations to view exhibits at cultural facilities with a guide. The program for this system comprises the following steps:

[2094] The server receives data (images, videos, text) about exhibits provided by cultural facilities and stores it as a digital archive. Specifically, the server uses image analysis technology (such as OpenCV) to extract detailed information and stores it in a database (such as SQLite). This makes it possible to manage metadata (such as the name of the work, artist, era, and description) for each exhibit.

[2095] Next, the server generates a 3D virtual space based on the digital archive. Exhibits are placed in this virtual space in the same way as they would be in a real cultural facility. For example, the server uses a 3D rendering engine such as Unity or Unreal Engine to recreate a virtual art gallery or museum. The server also manages the placement data of the exhibits and places them in the appropriate locations.

[2096] Next, the guide enters and registers their profile information (career history, expertise, achievements) and areas of expertise into the server using a dedicated application. This saves and manages the guide information in a database. For example, if a guide enters their profile as a "historian," this information is saved in the system.

[2097] Users can create an account through the application and enter their interests and desired exhibits. They can also enter the time they are available. This information is sent to the server and analyzed. The server then matches users with the most suitable guide based on their interests. For example, if a user expresses an interest in "Renaissance paintings," the server will select a guide with that expertise.

[2098] At the designated date and time, the guide and user access the application. The guide will guide the user through the virtual space and provide real-time commentary. Users can send questions and comments about the exhibits in real time via their smartphones or tablets. This two-way communication function allows users to view the exhibits while communicating directly with the guide.

[2099] After the tour ends, feedback is collected from users. The server analyzes the collected feedback and provides it to guides and cultural facilities. The generative AI model also analyzes the feedback and suggests improvements to guide behavior and optimal placement of exhibits. Furthermore, by inputting prompts such as "Generate optimal guide suggestions based on the user's interests and feedback" into the generative AI model, the selection of the optimal guide can be automated.

[2100] Regarding fee collection, tour fees are collected from users and distributed to guides, cultural facilities, and system operators. For example, fees can be collected using credit cards or electronic payment services (such as PayPal), and the system automatically distributes them to each party.

[2101] As a specific example, if User A uses the app and inputs that he or she is interested in Renaissance paintings, the server will match the user with Guide B, who has specialized knowledge, based on that information. The user accesses the app at the specified date and time, and Guide B will guide the user through the virtual museum. After the tour ends, User A provides feedback, which is saved on the server. The generative AI model then analyzes the feedback and suggests areas for improving Guide B's explanation and optimal placement of exhibits.

[2102] Example prompt: "Generate optimal guide suggestions based on the user's interests and feedback."

[2103] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2104] Step 1: Receive exhibit data from cultural facilities and store it as a digital archive

[2105] The server receives images, videos, and text data of exhibits provided by cultural facilities via an API. This data is sent to the server, where image analysis technology (e.g., OpenCV) is used to extract detailed information (such as the name of the work, the artist, the year, and a description) and store it in a database (e.g., SQLite). The input is data about the exhibits, and the output is digitally archived data.

[2106] Step 2: Generate a 3D virtual space based on the digital archive

[2107] The server generates a virtual space using a 3D rendering engine such as Unity or Unreal Engine based on the stored digital archive. Exhibits are placed in appropriate positions within the virtual space. The input is digital archive data, and the output is a 3D virtual space.

[2108] Step 3: Enter and manage your guide's profile information and specialty information

[2109] The guide uses a terminal to input his / her profile information (career history, expertise, achievements) and areas of expertise. This information is sent to the server and stored in a database. The input is the guide's profile information, and the output is the guide information stored in the database.

[2110] Step 4: Enter your interests and desired exhibits

[2111] Users create an account using their device and enter their interests (the exhibits and themes they wish to view) and the time they have available. This information is sent to the server and analyzed. The input is the user's interests and desired exhibits, and the output is the analyzed user information.

[2112] Step 5: Matching the best guide with the user

[2113] The server performs appropriate matching based on the input guide information and user information. Using a generative AI model, it inputs a prompt such as "Generate optimal guide suggestions based on the user's interests and feedback" and selects the optimal guide. The input is guide information and user information, and the output is the matched guide and user.

[2114] Step 6: A guide will guide you through the virtual space

[2115] At the specified date and time, the guide and user access the application using their terminals. The guide will guide the user through the virtual space in real time and provide explanations of the exhibits. The input is the guide and user matched with the specified date and time, and the output is the real-time guided tour.

[2116] Step 7: Send your questions and comments via interactive communication

[2117] Visitors can send questions and comments about the exhibits in real time via smartphones or tablet devices, and the guide will respond accordingly. The input is the visitor's questions and comments, and the output is the guide's responses.

[2118] Step 8: Collect and analyze feedback

[2119] After the tour ends, feedback is collected from the user. The server analyzes the collected feedback and provides it to the guide and cultural facilities. It also analyzes the feedback using a generative AI model and proposes improvements to the guide's behavior and optimal placement of exhibits. The input is the user's feedback, and the output is the analysis results and proposals.

[2120] Step 9: Toll collection and distribution

[2121] The server collects tour fees from users and distributes them to guides, cultural facilities, and system operators using credit card payments or electronic payment services (such as PayPal). The input is the user's payment information, and the output is the distributed fee.

[2122] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2123] The system of the present invention allows remote users to view exhibits at cultural facilities with a guide, and also combines an emotion engine that recognizes the user's emotions. The system's program consists of many modules and methods, which provide users with a high-quality cultural experience and promote the improvement of guides and cultural facilities.

[2124] The main components of this system are as follows:

[2125] Digital archiving of art museums

[2126] server:

[2127] Images, videos, and text data of exhibits are received from cultural facilities via API. This data is stored on a server, and metadata (such as the name of the work, artist, year, and description) for each exhibit is analyzed and saved as a digital archive. For example, image data of a "famous painting" is received, and detailed information is extracted using image analysis technology and saved in the digital archive.

[2128] Creating a virtual museum

[2129] server:

[2130] The server generates a three-dimensional virtual space based on the stored digital archives. Within this virtual space, exhibits are arranged in the same way as they would be in a real cultural facility. For example, the interior of a "historical building" can be recreated, with "ancient artifacts" on display placed in appropriate locations.

[2131] Guide registration and management

[2132] Terminal (Guide):

[2133] Guides use a dedicated application to register their profile information (career history, expertise, achievements) and areas of expertise. This information is sent to the server and stored in a database. For example, a guide can enter a profile as a "historian," and that information is stored and managed within the system.

[2134] User registration and matching system

[2135] Terminal (user):

[2136] Users create an account through the application and enter their interests (the exhibits and themes they wish to view) and available time. The server matches them with the most suitable guide based on the information they enter. For example, if a user expresses an interest in "Renaissance paintings," the server will select a guide who is knowledgeable about Renaissance paintings.

[2137] Tour execution and feedback collection

[2138] Terminals (Guide and User):

[2139] At the specified date and time, the guide and user access the application. The guide guides the user through the virtual museum and provides real-time commentary. Users can share questions and comments through chat or voice communication. For example, while the guide explains "modern sculpture" to the user, the user can freely ask questions and provide feedback after the explanation.

[2140] server:

[2141] The server receives feedback from users and stores it in a database. The server analyzes the feedback and provides detailed reports to guides and cultural facilities.

[2142] Incorporating an emotion engine

[2143] Terminal (user):

[2144] The system uses an emotion engine that analyzes the user's facial expressions, tone of voice, etc. to monitor the user's emotional state in real time. For example, if the user feels surprise or excitement after seeing an exhibit, that emotional state will be analyzed.

[2145] server:

[2146] The emotion engine analyzes the user's emotional data and provides feedback to the guide based on that information, such as instructing the guide to provide a more detailed explanation of an exhibit that the user is interested in.

[2147] Revenue Share Distribution

[2148] server:

[2149] The server collects the tour fee from the user and distributes it to the guide, the cultural facility, and the system operator. For example, if the tour fee is paid by credit card, the server automatically distributes the fee to each party.

[2150] By integrating these functions, this system simultaneously digitizes cultural facilities, improves user convenience, and increases revenue. Furthermore, by incorporating an emotion engine, it can enrich the user experience and provide data useful for improving the operation of guides and cultural facilities. This is expected to promote financial support and digitalization for cultural facilities, and improve user cultural satisfaction.

[2151] The processing flow will be explained below.

[2152] Digital archiving of art museums

[2153] Step 1:

[2154] server:

[2155] Images, videos, and text data provided by cultural facilities are received via API.

[2156] Step 2:

[2157] server:

[2158] Store the received data in local storage or cloud storage.

[2159] Step 3:

[2160] server:

[2161] Metadata (title, artist, year, description) is extracted from the stored data using an image analysis algorithm.

[2162] Step 4:

[2163] server:

[2164] The extracted metadata is registered in a database.

[2165] Creating a virtual museum

[2166] Step 1:

[2167] server:

[2168] Using 3D modeling tools, a three-dimensional virtual space is generated based on the digital archive.

[2169] Step 2:

[2170] server:

[2171] The coordinate data of the exhibits is set within the generated virtual space.

[2172] Step 3:

[2173] server:

[2174] Based on the coordinate data, the exhibits are arranged and interactive functions are added.

[2175] Guide registration and management

[2176] Step 1:

[2177] Terminal (Guide):

[2178] The guide opens a dedicated application and enters their profile information and areas of expertise.

[2179] Step 2:

[2180] Terminal (Guide):

[2181] Add details about yourself, your achievements, etc.

[2182] Step 3:

[2183] server:

[2184] Receives guide input information and stores it in a database.

[2185] User registration and matching system

[2186] Step 1:

[2187] Terminal (user):

[2188] The user opens the dedicated application and creates an account.

[2189] Step 2:

[2190] Terminal (user):

[2191] Enter your interests, desired exhibits, and available time.

[2192] Step 3:

[2193] server:

[2194] It takes the user's input information and searches the database for the appropriate guide.

[2195] Step 4:

[2196] server:

[2197] Using an AI algorithm, it matches the user's interests, the guide's areas of expertise, and exhibit information.

[2198] Step 5:

[2199] server:

[2200] The matching results are presented to the user, and the user selects a guide.

[2201] Tour execution and feedback collection

[2202] Step 1:

[2203] Terminals (Guide and User):

[2204] The Guide and user access the application at the specified date and time.

[2205] Step 2:

[2206] Terminal (Guide):

[2207] A guide will guide you through the virtual museum and provide real-time commentary on the exhibits.

[2208] Step 3:

[2209] Terminal (user):

[2210] Users share questions and thoughts through chat functionality and voice communication.

[2211] Step 4:

[2212] Terminal (user):

[2213] After the tour, the user fills out a feedback form.

[2214] Step 5:

[2215] server:

[2216] Receive user feedback and store it in a database.

[2217] Step 6:

[2218] server:

[2219] The feedback data is analysed and provided to guides and cultural institutions as detailed reports.

[2220] Incorporating an emotion engine

[2221] Step 1:

[2222] Terminal (user):

[2223] Activate an emotion engine that analyzes the user's facial expressions and tone of voice.

[2224] Step 2:

[2225] Terminal (user):

[2226] The emotion engine analyzes the user's emotional state in real time and sends the data to the server.

[2227] Step 3:

[2228] server:

[2229] It receives emotional data sent in real time and generates feedback to the guide based on the analysis results.

[2230] Step 4:

[2231] Terminal (Guide):

[2232] The guide receives real-time feedback and adjusts the content and progress of the explanation based on the user's emotional state.

[2233] Revenue Share Distribution

[2234] Step 1:

[2235] server:

[2236] The tour fee is collected from the user via credit card or electronic payment.

[2237] Step 2:

[2238] server:

[2239] Temporarily store the collected fees.

[2240] Step 3:

[2241] server:

[2242] The fees will be distributed to guide, cultural institution and platform accounts based on a revenue share percentage.

[2243] Step 4:

[2244] server:

[2245] Process payments to each distribution recipient.

[2246] Through these processing steps, the system of the present invention simultaneously digitizes cultural facilities, improves user convenience, and increases revenue. Furthermore, by incorporating an emotion engine, it can enrich the user experience and provide data that is useful for improving the operation of guides and cultural facilities.

[2247] Example 2

[2248] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2249] When remote visitors view exhibits at cultural facilities, they need a high-quality, immersive experience while also being able to effectively collect, manage, and utilize their feedback on guides and cultural facilities. A system is also needed that can grasp the emotional state of visitors in real time and enable guides to respond appropriately based on that information. Fair and efficient distribution of revenue is also required.

[2250] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2251] In this invention, the server includes means for receiving exhibit data provided by the cultural facility and saving it as a digital archive, means for generating a three-dimensional virtual space based on the digital archive and appropriately arranging the exhibits within the virtual space, and means for inputting and managing guide profile information and specialty information. This allows remote users to enjoy a high-quality, immersive cultural experience while also enabling monitoring of the users' emotional states and real-time responses, and realizing fair and efficient distribution of revenue.

[2252] "Exhibits" refers to artworks, historical artifacts, artifacts, and other cultural assets that are made available to the public at cultural institutions.

[2253] A "guide" refers to a person who has knowledge of the exhibits at a cultural facility and provides explanations to visitors.

[2254] "User" refers to an individual in a remote location who wishes to view the exhibits at a cultural facility.

[2255] A "digital archive" refers to a system that stores and manages exhibit data (images, videos, text, etc.) in digital format.

[2256] "Three-dimensional virtual space" refers to a three-dimensional virtual space generated using digital technology that recreates the interior of an actual cultural facility.

[2257] "Profile information" refers to information about the guide, such as their background, expertise, and achievements.

[2258] "Emotional state" refers to emotions (surprise, excitement, joy, etc.) analyzed from the user's facial expressions and tone of voice.

[2259] "Feedback" refers to impressions, opinions, and ratings collected from users regarding exhibits and guides.

[2260] "Distribution" refers to the fair and efficient division of revenues (e.g., tour fees) among guides, cultural institutions, and system operators.

[2261] The system of the present invention allows remote users to view exhibits at cultural facilities with a guide, and also combines an emotion engine that recognizes the user's emotions. The system is composed of multiple modules and means, providing users with a high-quality cultural experience and promoting the improvement of guides and cultural facilities.

[2262] Digital archiving of art museums

[2263] server:

[2264] Images, videos, and text data of exhibits are received from cultural facilities via API. For example, TensorFlow is used as a specific image analysis technology to extract the name of the work and information about the artist from the received image data. This data is stored on a server as metadata (work name, artist, year, description, etc.) and managed as a digital archive. As a specific example, image data of a famous painting is received, and the name of the work and information about the artist are extracted using image analysis technology.

[2265] Creating a virtual museum

[2266] server:

[2267] A 3D virtual space is generated based on the stored digital archives. 3D models of the exhibits are created using 3D engines such as Unity or Unreal Engine, and then placed inside the virtual museum. Specifically, the interior of a "historical building" is recreated, and the "ancient artifacts" to be displayed there are placed in the appropriate locations.

[2268] Guide registration and management

[2269] Terminal (navigator):

[2270] Guides use a dedicated application to register their profile information (career history, expertise, achievements) and areas of expertise. This information is sent to the server and stored in a database. For example, if a guide enters their profile as a "historian," the information is sent to the server in real time and stored in the database.

[2271] User registration and matching system

[2272] Terminal (user):

[2273] Users create an account through the application and input their interests (the exhibits and themes they wish to view) and available time. The server matches them with the most suitable guide based on the information they input. For example, if a user expresses an interest in "Renaissance paintings," the server will select a guide who is knowledgeable about Renaissance paintings.

[2274] Tour execution and feedback collection

[2275] Terminals (instructor and user):

[2276] At the designated date and time, the guide and user access the application. The guide will guide the user through the virtual museum and provide real-time commentary. Users can share their questions and impressions through chat and voice communication. For example, while the guide is explaining about "modern sculpture," users can freely ask questions and provide feedback after the explanation.

[2277] server:

[2278] Feedback from users is received and stored in a database. The server analyzes the feedback and provides detailed reports to guides and cultural facilities.

[2279] Incorporating an emotion engine

[2280] Terminal (user):

[2281] The system uses an emotion engine that analyzes the user's facial expressions and tone of voice to monitor the user's emotional state in real time. For example, the system uses the emotion recognition API from Microsoft Azure Cognitive Services as a specific analysis technology. For example, if a user feels surprise or excitement when looking at an exhibit, that emotional state is analyzed.

[2282] server:

[2283] The emotion engine analyzes the user's emotional data and provides feedback to the guide based on that information. For example, it can instruct the guide to provide a more detailed explanation of an exhibit that the user is interested in.

[2284] Revenue Share Distribution

[2285] server:

[2286] The server collects the tour fee from the user and distributes it to the guide, the cultural facility, and the system operator. Specifically, it uses a payment gateway API such as Stripe for payment processing. When the tour fee is paid by credit card, the server automatically distributes the fee to each party.

[2287] Specific examples and examples of prompts for generative AI models

[2288] As a concrete example, consider a case where a user wants to view a "famous Renaissance painting." In this case, the server selects a "guide with expertise in Renaissance painting." If the emotion engine detects the user's excitement, it instructs the guide to provide more detailed information about the exhibit. An example of a prompt sentence for the generative AI model is as follows:

[2289] 1. "User is interested in Renaissance paintings. Please select relevant guides."

[2290] 2. "The user seems surprised by the exhibit. Please instruct the guide to provide a detailed explanation."

[2291] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2292] Step 1:

[2293] The server receives and stores data from the cultural institution.

[2294] Input: Images, videos, and text data of exhibits provided by cultural facilities via API

[2295] Specific operation: The server receives data on the exhibits using an API. To do this, it uses image analysis technology such as TensorFlow to extract the name of the work and information about the artist from the received image data and saves it as metadata.

[2296] Output: The analyzed metadata and digital data of the exhibits are saved on the server.

[2297] Step 2:

[2298] The server generates a three-dimensional virtual space.

[2299] Input: Stored digital archive data

[2300] How it works: The server uses a 3D engine such as Unity or Unreal Engine to generate a 3D virtual s...

Claims

1. A system that enables a user located remotely to view exhibits at a cultural facility with a guide, comprising: A means to receive data on exhibits provided by cultural facilities and store it as a digital archive; A means for generating a three-dimensional virtual space based on the digital archive and appropriately arranging exhibits within the virtual space; A means for inputting and managing guide profile information and specialty information; A means for inputting and displaying the user's interests and desired exhibit contents; A means for matching the most suitable guide with the user based on the input guide information and user information; A guide will lead you through the virtual space and provide real-time commentary; A means of collecting and analyzing user feedback; a means of providing collected feedback to guides and cultural institutions; A means for collecting tour fees from users and distributing the fees to guides, cultural institutions, and system operators; A system including:

2. 2. The system according to claim 1, further comprising an interactive function added to the generated three-dimensional virtual space, which allows users to obtain information about the exhibits in a two-way manner.

3. 2. The system according to claim 1, further comprising a function of converting the guide's commentary into text in real time, translating it into multiple languages, and displaying it to the user as subtitles.

4. The system according to claim 1, further comprising means for suggesting improvements to the guide and cultural facilities based on the analysis results of the feedback, and reflecting the improvements in future exhibition management.

5. The system according to claim 1, further comprising a platform for online sales of merchandise related to exhibits at cultural facilities, and a means for increasing revenues related to the exhibits.

Images