system

Abstract

A system is provided.SOLUTION: A system comprising means for obtaining profile data of a user, means for using a generative AI to analyze the user's artistic preferences based on the obtained profile data, means for rendering an artistic work generated by the generative AI into a VR space, means for obtaining data in real-time based on eye tracking or gesture recognition of the user, means for analyzing the obtained data and using the generative AI to evolve the artistic work, and means for updating and reflecting the evolved artistic work into the VR space.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Traditional art exhibition methods are static and fixed, making it difficult to customize to meet individual user preferences and responses. They also require users to physically visit galleries or museums, which is not suited to the digital and remote era. This limits access to certain demographics. Furthermore, traditional educational methods are limited in their digital art education methods and are insufficient to stimulate cultural learning interest among young people. [Means for solving the problem]

[0005] In order to solve the above problems, we have developed a system that provides the following means.

[0006] 1. Provide a means to obtain user profile data.

[0007] 2. Providing a means to analyze users' art preferences using generative AI based on their profile data.

[0008] 3. Based on the analysis results, generative AI will generate artwork and provide a means to render it in VR space.

[0009] 4. Provide a means to acquire user response data in real time through eye tracking and gesture recognition, and analyze this data to evolve artworks.

[0010] 5. Provide a means to instantly update and reflect evolved artworks in VR space.

[0011] Furthermore, by providing a means to store the user's gaze tracking, gesture recognition, and voice input data and provide it to the generation AI for optimizing the next exhibition, as well as a means to collect feedback from users on the evolved artwork and store it in the generation AI's learning database, it will be possible to realize individually optimized interactive art exhibitions.

[0012] "User Profile Data" means data that allows us to identify or personalize you, such as your personal information, preferences, and past interaction data.

[0013] "Generative AI" refers to artificial intelligence that uses machine learning algorithms to generate new artworks based on given data and conditions.

[0014] "VR space" is a digital simulation that uses virtual reality technology to allow users to experience it visually and aurally.

[0015] "Eye tracking" is a technology that uses sensors to detect the movement of a user's eyes and collect that data.

[0016] "Gesture recognition" is a technology that captures the user's hand and body movements using sensors and analyzes them as data.

[0017] "Rendering" is the process of displaying a generated artwork in a virtual space using a 3D engine or similar.

[0018] "Real-time" refers to processing occurring immediately at the moment a user action or response occurs.

[0019] "Feedback" means the act or result of a user providing a rating or comment on their experience.

[0020] A "learning database" is a collection of accumulated data that generative AI uses to continuously learn.

[0021] "Evolution" refers to the process by which the generated artwork changes based on user responses, incorporating new forms and elements. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram illustrating a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0023] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0024] First, the terms used in the following description will be explained.

[0025] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0026] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0027] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0028] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0029] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0030] [First embodiment]

[0031] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0032] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0033] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0034] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0035] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0036] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0037] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0038] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0039] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0040] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0041] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0042] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0043] This invention relates to a system for providing an interactive virtual reality (VR) space in which artwork is generated and evolved according to the user's preferences and reactions. Specific aspects of the system are described below.

[0044] User Awareness and Initial Setup

[0045] Server: When a user logs into the system, the server retrieves the user's profile data, which includes the user's name, age, art preferences, past reaction data, etc.

[0046] Device: Detects when the user puts on a VR headset and prepares the VR environment, including calibrating the VR equipment and configuring it to understand the user's position in the environment.

[0047] User: A user puts on a VR headset and controllers and begins accessing the system. First-time users are prompted to enter basic profile information.

[0048] Analysis of user preferences

[0049] Server: Analyzes user profile data and uses generative AI to analyze the user's art preferences. This process uses machine learning algorithms to identify trends in the user's preferred art styles (abstract, figurative, color patterns, etc.).

[0050] Server: The generative AI generates the initial artwork based on the analysis results. The data of this generated artwork is converted into a VR-compatible format.

[0051] Interactive VR space generation

[0052] Terminal: A VR space is constructed based on the artwork data sent from the server, providing a 3D interface that responds to the user's visual and auditory senses.

[0053] Device: Captures the user's actions in the VR space (eye tracking, gesture recognition, voice input, etc.) in real time and sends the data to the server.

[0054] Real-time reactions and artistic evolution

[0055] Server: The generative AI analyzes user reaction data and uses it to evolve the artwork, highlighting areas that the user particularly found interesting.

[0056] Server: Data of the evolved artwork is sent to the device, and is instantly reflected in the VR space.

[0057] Device: As new artworks are generated, existing artworks are updated in real time, completely transforming the user experience.

[0058] Feedback accumulation and optimization

[0059] Server: Collects user feedback and ratings and stores them in the generative AI's learning database. This feedback data is used to optimize the next exhibit.

[0060] Terminal: After the exhibition ends, a feedback form will be displayed to the user, providing an interface where they can enter their ratings and comments.

[0061] Specific examples

[0062] Example 1: When a new user accesses the system and enters their profile information, the generative AI generates an abstract painting in the VR space based on the user's preferences. If the user focuses their gaze on a particular color or shape, the artwork evolves to emphasize that aspect.

[0063] Example 2: When a repeat user revisits the system, artworks are generated that are more tailored to the user based on their past data, resulting in a more refined and personalized art experience for the user.

[0064] As described above, this system combines generative AI and VR technology to create dynamic art exhibits that respond to the user's preferences and reactions. By utilizing digital space, it can provide new art experiences that transcend physical constraints, and can also make a significant contribution to the fields of education and culture.

[0065] The processing flow will be explained below.

[0066] Step 1:

[0067] User: Accesses and logs into the system.

[0068] Specific operation: The user enters their ID and password and presses the login button.

[0069] Step 2:

[0070] Server: Retrieves the user's profile data.

[0071] Specific operation: Search and extract the user's past art viewing history, preferences, rating data, etc. from the database.

[0072] Step 3:

[0073] Device: Detects when the user is wearing a VR headset and configures the environment.

[0074] Specific operation: Calibrate the VR equipment and obtain the user's body position information.

[0075] Step 4:

[0076] Server: Based on the acquired profile data, the server uses generative AI to analyze the user's artistic preferences.

[0077] What it does: It feeds profile data into machine learning algorithms to identify user preferences.

[0078] Step 5:

[0079] Server: Generative AI generates an initial artwork based on the analysis results.

[0080] What it does: The AI ​​model generates digital art by combining appropriate art styles and elements, then converts the data into a VR-compatible format.

[0081] Step 6:

[0082] Server: Sends the generated artwork data to the device.

[0083] Specific operation: Digital art data is compressed and sent to the terminal using a high-speed communication protocol.

[0084] Step 7:

[0085] Terminal: Generates a VR space based on the received art data.

[0086] Specific operation: Using a 3D rendering engine, artwork is displayed in a VR space.

[0087] Step 8:

[0088] User: Experience the artwork in a VR space.

[0089] Specific actions: Users interact with the artwork using eye tracking and gestures.

[0090] Step 9:

[0091] Device: Sends real-time reaction data from the user's gaze tracking and gesture recognition to the server.

[0092] Specific operation: Data acquired by the gaze tracking sensor and gesture recognition system is analyzed and sent to the server in real time.

[0093] Step 10:

[0094] Server: Analyzes user reaction data, and the generative AI uses this data to evolve the artwork.

[0095] Specific behavior: Based on real-time data, the system makes changes to the artwork, such as adding new elements or emphasizing parts that users like.

[0096] Step 11:

[0097] Server: The data of the evolved artwork is sent back to the device.

[0098] Specific behavior: Sends updated art data at high speed.

[0099] Step 12:

[0100] Device: Evolved artworks are reflected in the VR space in real time.

[0101] What it does: Instantly update artwork in the VR space and display new content to users.

[0102] Step 13:

[0103] User: After completing the exhibition experience, enter your evaluation in the feedback form.

[0104] Specific action: Fill out the feedback form that appears and provide your rating and comments about the art experience, then submit it.

[0105] Step 14:

[0106] Server: Stores user feedback data and provides it to the generation AI for next display optimization.

[0107] Specific operation: Feedback data is accumulated in a database and used as learning data for the generative AI.

[0108] Example 1

[0109] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0110] Conventional art exhibition systems have struggled to dynamically generate and evolve artworks based on individual user preferences and real-time responses. They also have limited means to effectively incorporate interactive data, such as user gaze tracking and gesture recognition. As a result, the experience for each user is not fully personalized, and there is no appropriate mechanism for incorporating user feedback into future exhibitions.

[0111] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0112] In this invention, the server includes means for acquiring user profile data, means for using artificial intelligence to analyze user preferences based on the acquired user profile data, means for rendering a digital artwork generated by the artificial intelligence in a virtual reality space, means for acquiring data based on the user's gaze tracking and motion recognition in real time, means for analyzing the acquired data and using artificial intelligence to evolve the digital artwork, and means for updating and reflecting the evolved digital artwork in the virtual reality space, thereby enabling a dynamic and personalized art experience that reflects individual user preferences and real-time reactions.

[0113] "User profile data" refers to any information related to a user who logs into the system, such as the user's name, age, art preferences, and past response data.

[0114] "Artificial intelligence" refers to technologies including machine learning algorithms and deep learning models that analyze user profile data and real-time reaction data to generate and evolve appropriate artworks.

[0115] "Digital artwork" refers to art generated by artificial intelligence, the data of which is converted into a format (e.g., FBX or OBJ) that can be rendered in a virtual reality space.

[0116] "Virtual reality space" refers to a three-dimensional digital environment that users can immerse themselves in using devices such as VR headsets.

[0117] "Eye tracking" refers to a technology that tracks the direction a user is looking or a specific object in a virtual reality space in real time.

[0118] "Motion recognition" refers to the technology of capturing a user's gestures and body movements with sensors and analyzing them.

[0119] "Real-time data" refers to data obtained through user gaze tracking and / or motion recognition that is processed and analyzed in real time.

[0120] MODE FOR CARRYING OUT THE INVENTION

[0121] This invention relates to a system that provides an interactive virtual reality (VR) space in which digital artwork is generated and evolved according to the preferences and reactions of users. Specific embodiments will be described below.

[0122] User Awareness and Initial Setup

[0123] Server: When a user logs into the system, the server retrieves the user's profile data (such as name, age, art preferences, past reaction data, etc.) This profile data is retrieved using database queries, and the data is cleansed and analyzed using Python libraries (e.g., Pandas, NumPy).

[0124] Device: When a user puts on a VR headset (e.g., a general-purpose VR headset), the device detects this information and prepares the VR environment, including calibrating the VR device and configuring it to understand the user's position (e.g., 6DOF tracking).

[0125] User: The user puts on the VR headset and controllers and begins accessing the system. First-time users are asked to enter basic profile information (such as name, age, and art preferences) through a form in the interface.

[0126] Analysis of user preferences

[0127] Server: The server analyzes the acquired user profile data and uses generative AI to analyze the user's art preferences. In this process, machine learning algorithms (e.g., TensorFlow, PyTorch) are used to identify trends in the user's preferred art styles (abstract art, figurative art, color patterns, etc.).

[0128] Server: Generative AI (e.g., GPT-3, GANs) generates the initial digital artwork based on the analysis results. The Assimp library is used to convert this generated artwork data into a VR-compatible format (e.g., FBX format or OBJ format).

[0129] Interactive VR space generation

[0130] Terminal: Using Unity or Unreal Engine, a VR space is created based on the artwork data sent from the server. This provides a 3D interface that responds to the user's vision and hearing.

[0131] Device: Configure it using the necessary APIs (e.g., OpenVR SDK, general-purpose VR SDK) to capture the user's actions in the VR space (e.g., eye tracking, motion recognition, voice input, etc.) in real time.

[0132] Real-time reactions and artistic evolution

[0133] Server: Analyzes the user's real-time reaction data (eye tracking data, motion recognition data, etc.), and the generative AI evolves the artwork based on this. Specifically, computer vision technology (e.g., OpenCV) is used to analyze the gaze data.

[0134] Server: Sends the evolved artwork data to the device. This process uses WebSocket to exchange data in real time.

[0135] Device: When a new artwork is generated, the existing artwork is instantly updated in the VR space, leveraging the real-time rendering capabilities of Unity and Unreal Engine, so the changes are instantly reflected in the user's vision.

[0136] Feedback accumulation and optimization

[0137] Server: After the exhibition, the feedback and evaluations collected from users are stored in the AI's learning database. This data will be used for future art generation.

[0138] Terminal: After the exhibition ends, a feedback form is displayed to the user, providing an interface where they can enter their ratings and comments. This form data is collected using HTML and JavaScript and sent to the server.

[0139] Specific examples

[0140] Example 1:

[0141] Server: When a new user logs in, it collects profile information and queries the database. Python is used to clean the data and extract preferences. The generative AI generates abstract art based on the data and converts it into FBX format.

[0142] Device: Receives artwork data and displays it in the VR space using Unity. When the user directs their gaze at a specific color or shape, eye-tracking data is sent to the server.

[0143] Server: To evolve the artwork based on the eye-tracking data, the generative AI updates the artwork design. It converts the new data into a VR format and sends it to the device.

[0144] Device: The new artwork is instantly updated in the VR space, allowing users to experience the changes in real time.

[0145] Example 2:

[0146] Server: When a repeat user accesses the site again, the initial settings are made based on the user's past profile data and responses, and an art piece is generated that is further tailored to the user.

[0147] Device: Compares existing art with new art and dynamically updates the VR space. When the user responds with gestures to specific shapes, the information is sent to the server in real time.

[0148] Server: The artwork is partially evolved based on gesture recognition data and the information is sent to the device.

[0149] On your device: Display updated artwork so that your art experience is more refined and personalized than the last time.

[0150] Prompt Sentence Examples

[0151] "Describe the process of generating abstract paintings based on user profile information and analyzing the user's eye-tracking data to evolve the artwork."

[0152] As described above, this system combines generative AI and VR technology to provide a dynamic art experience that responds to the user's preferences and reactions. By utilizing digital space, it can provide new art experiences that transcend physical constraints and make significant contributions to the fields of education and culture.

[0153] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0154] Program processing flow

[0155] Step 1: User Awareness and Initial Setup

[0156] Server: When a user logs in to the system, the server retrieves the user's profile data (such as name, age, art preferences, and past reaction data) using a database query and cleanses the data using Python libraries (e.g., Pandas and NumPy).

[0157] Input: User login information.

[0158] Output: Cleansed user profile data.

[0159] Device: When the user puts on the VR headset, the device detects this information and prepares the VR environment. Specifically, it calibrates the VR equipment and sets up the user's position (for example, 6DOF tracking).

[0160] Input: VR headset wearing information.

[0161] Output: Calibrated VR environment setup.

[0162] User: The user puts on the VR headset and controllers and begins accessing the system. First-time users enter basic profile information (such as name, age, and art preferences) through a form in the interface.

[0163] Input: Basic profile information.

[0164] Output: The entered profile information.

[0165] Step 2: Analyzing user preferences

[0166] Server: The server analyzes user profile data and uses generative AI (e.g., TensorFlow or PyTorch) to analyze the user's art preferences. It uses machine learning algorithms to identify trends in the user's preferred art styles (e.g., abstract, figurative, color patterns, etc.).

[0167] Input: Cleansed user profile data.

[0168] Output: User's art preference information.

[0169] Server: The generative AI generates the initial digital artwork based on the analysis results. The generated artwork data is converted into a VR-compatible format (e.g., FBX or OBJ format) using the Assimp library.

[0170] Input: User's art preference information.

[0171] Output: Digital artwork data in a VR-compatible format.

[0172] Step 3: Creating an interactive VR space

[0173] Device: Using Unity or Unreal Engine, a VR space is created based on the artwork data sent from the server, providing a 3D interface that responds to the user's visual and auditory senses.

[0174] Input: Digital artwork data in a VR-compatible format.

[0175] Output: The constructed VR space.

[0176] Device: Captures the user's actions in the VR space (eye tracking, motion recognition, voice input, etc.) in real time and sends the data to the server. Uses the required API (e.g., OpenVR SDK, general-purpose VR SDK).

[0177] Input: User operation data.

[0178] Output: Operation data sent to the server.

[0179] Step 4: Real-time reactions and artistic evolution

[0180] Server: Analyzes the user's real-time reaction data (eye tracking data, motion recognition data, etc.), and the generative AI evolves the artwork based on this data. Computer vision technology (e.g., OpenCV) is used to analyze the gaze data.

[0181] Input: Real-time user response data.

[0182] Output: Evolved digital artwork data.

[0183] Server: Sends the evolved artwork data to the device. Data is exchanged in real time using WebSocket.

[0184] Input: Evolved digital artwork data.

[0185] Output: Artwork data sent to the device.

[0186] Device: As new artworks are generated, existing artworks are updated in real time within the VR space, utilizing the real-time rendering capabilities of Unity and Unreal Engine.

[0187] Input: Evolved digital artwork data.

[0188] Output: Updated VR space.

[0189] Step 5: Gather feedback and optimize

[0190] Server: After the exhibition is over, the feedback and evaluations collected from users are stored in the Generative AI's learning database. This data is used to optimize the next exhibition.

[0191] Input: User feedback data.

[0192] Output: Saved feedback data.

[0193] Terminal: After the exhibition ends, a feedback form is displayed to the user, providing an interface where they can enter their ratings and comments. The form data is collected using HTML and JavaScript and sent to the server.

[0194] Input: User ratings and comments.

[0195] Output: The feedback form data sent to the server.

[0196] Through these processing steps, the system provides a dynamic art experience that responds to user preferences and real-time responses.

[0197] (Application example 1)

[0198] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0199] Conventional entertainment systems for autonomous vehicles have the drawback of being difficult to provide content that responds to individual user preferences and real-time responses, and are difficult to provide an experience with enhanced visual and tactile interactivity due to the limited environment inside the vehicle.

[0200] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0201] In this invention, the server includes means for acquiring user profile data, means for using a generation AI to analyze the user's art preferences based on the acquired profile data, means for rendering artwork generated by the generation AI in a virtual reality space, means for acquiring data based on the user's gaze tracking and gesture recognition in real time, means for analyzing the acquired data and using the generation AI to evolve the artwork, means for updating and reflecting the evolved artwork in the virtual reality space, and means for providing a terminal for using these means within an autonomous vehicle, thereby enabling the provision of a high-quality, personalized art experience in real time even within an autonomous vehicle.

[0202] "User Profile Data" means data that includes personal information about you, such as your name, age, and artistic preferences.

[0203] "Generative AI" is artificial intelligence that uses machine learning algorithms to generate and evolve artworks based on user preferences.

[0204] A "virtual reality space" is a three-dimensional virtual space that provides users with an immersive experience through visual and auditory interfaces.

[0205] "Eye tracking" is a technology that detects and analyzes the user's eye movements in real time.

[0206] "Gesture recognition" is a technology that uses cameras and sensors to detect and analyze the movements of a user's hands and fingers.

[0207] "Real-time acquisition means" refers to a system or device that instantly collects interaction data such as user gaze and gestures.

[0208] An "autonomous vehicle" is a vehicle that drives autonomously using artificial intelligence and sensor technology.

[0209] A "terminal" is a device or system (e.g., a head-mounted display) that allows a user to access a virtual reality space.

[0210] A "feedback form" is an interface that allows users to enter their opinions and evaluations of artworks.

[0211] The "generative AI learning database" is a database used to accumulate user feedback and reaction data and improve the performance of the generative AI.

[0212] The embodiments of the present invention will be specifically described below.

[0213] Retrieving a user profile

[0214] When a user logs into the system, the server retrieves the user's profile data, which includes the user's name, age, art preferences, past interaction data, etc. This provides the basis for collecting data to provide a personalized experience for each user.

[0215] User Awareness and Initial Setup

[0216] The device detects when the user is wearing a VR headset and prepares the VR environment, including calibrating the head-mounted display (e.g., Oculus Quest) and configuring it to understand the user's position. When a user first accesses the system, they are required to enter basic profile information.

[0217] Art taste analysis

[0218] The server analyzes the profile data and uses generative AI to analyze the user's art preferences. This process uses machine learning algorithms to identify trends in the user's preferred art styles (abstract, figurative, color patterns, etc.).

[0219] Generating and Rendering Initial Artwork

[0220] The generative AI generates an initial artwork based on the user's preferences. The data for this artwork is converted into a format compatible with the virtual reality space and sent to the device. The device then creates a virtual reality space based on the data sent from the server. This provides a 3D interface that responds to the user's visual and auditory senses.

[0221] Interactive operation and real-time updates

[0222] The device captures the user's actions in the virtual reality space (eye tracking, gesture recognition, voice input, etc.) in real time and sends the data to the server. The server analyzes the user's reaction data, and the generative AI uses this data to evolve the artwork. Changes are made, such as emphasizing areas that the user has shown particular interest in. Data on the evolved artwork is sent to the device and instantly reflected in the virtual reality space, completely transforming the user's experience.

[0223] Feedback collection and optimization

[0224] The server collects user feedback and ratings and stores them in the generative AI's learning database. This feedback data is used to optimize the next exhibit. After the exhibit is over, the terminal displays a feedback form to the user, providing an interface where they can enter ratings and comments.

[0225] Use in self-driving vehicles

[0226] This system is installed in autonomous vehicles, allowing users to enjoy art experiences using VR headsets while traveling. The in-car device captures the user's actions in real time and works with a server to evolve the artwork, providing immersive entertainment even while traveling.

[0227] Examples:

[0228] Example 1: A new user accesses the system and enters their profile information. Generative AI generates an abstract painting in a virtual reality space based on the user's preferences. If the user focuses their gaze on a particular color or shape, the artwork evolves to emphasize that feature.

[0229] Example 2: When a returning user revisits the system, the system generates more personalized artwork based on their past data, resulting in a more refined and personalized art experience.

[0230] Example prompt sentence:

[0231] "Develop an app for autonomous vehicles that allows passengers to use VR headsets to experience and evolve art according to their preferences."

[0232] In this way, this invention combines generative AI and virtual reality technology to create dynamic art exhibits that respond to user preferences and reactions, providing a new entertainment experience even inside autonomous vehicles.

[0233] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0234] Step 1:

[0235] When a user logs in to the system, the server retrieves the user's profile data. The profile data includes the user's name, age, art preferences, and past response data. This provides the basis for the user to receive a personalized experience. The input is the user's login information, and the output is the user's profile data.

[0236] Step 2:

[0237] The device detects when the user puts on a VR headset and prepares the virtual reality environment. This includes calibrating the head-mounted display (e.g., Oculus Quest) and determining the user's position. The input is the state of the VR headset and user position information, and the output is information that the virtual reality environment is ready.

[0238] Step 3:

[0239] The server analyzes the acquired profile data and uses generative AI to analyze the user's art preferences. Machine learning algorithms are used to identify trends in the user's preferred art styles (abstract, figurative, color patterns, etc.). The input is the profile data, and the output is the analysis of the user's art preferences.

[0240] Step 4:

[0241] The server uses a generative AI to generate an initial artwork based on the user's preferences, converts it into a format compatible with the virtual reality space, and sends it to the device. The input is the user's art preference analysis results, and the output is VR-compatible initial artwork data.

[0242] Step 5:

[0243] The device constructs a virtual reality space based on the artwork data received from the server, providing a 3D interface that responds to the user's visual and auditory senses. The input is the initial artwork data, and the output is the constructed virtual reality space.

[0244] Step 6:

[0245] The device captures the user's actions in the virtual reality space (eye tracking, gesture recognition, voice input) in real time and sends the data to the server. The input is the user's action data, and the output is the data sent to the server.

[0246] Step 7:

[0247] The server analyzes the user's operation data, and the generative AI evolves the artwork based on this data. In particular, it makes changes such as emphasizing the parts that the user showed interest in. The input is the user's operation data, and the output is the evolved artwork data.

[0248] Step 8:

[0249] The server sends the evolved artwork data to the device, which then instantly reflects it in the virtual reality space. The input is the evolved artwork data, and the output is the updated virtual reality space.

[0250] Step 9:

[0251] The server collects user feedback and ratings and stores them in the generative AI's learning database. This feedback data is used to optimize the next display. The input is user feedback, and the output is an updated learning database.

[0252] Step 10:

[0253] The terminal displays a feedback form to the user after the exhibition has ended, providing an interface where users can enter their ratings and comments. The input triggers the end of the exhibition, and the output is the feedback information from the user.

[0254] The entire process enables a personalized art experience to be delivered in real time within the autonomous vehicle, and the system dynamically adapts to the user's preferences and reactions, delivering high-quality, immersive entertainment.

[0255] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0256] This invention relates to a system for providing an interactive virtual reality (VR) space that generates and evolves artwork by recognizing a user's preferences, reactions, and emotions. Specific aspects of the system are described below.

[0257] User Awareness and Initial Setup

[0258] Server: When a user logs into the system, the server retrieves the user's profile data, which includes the user's name, age, art preferences, past reaction data, etc.

[0259] Device: Detects when the user puts on a VR headset and configures the environment, including calibrating the VR equipment and locating the user in the environment.

[0260] User: A user puts on a VR headset and controllers and begins accessing the system. First-time users are prompted to enter basic profile information.

[0261] Analysis of user preferences

[0262] Server: Analyzes user profile data and uses generative AI to analyze the user's art preferences. This process uses machine learning algorithms to identify trends in the user's preferred art styles (abstract, figurative, color patterns, etc.).

[0263] Server: The generative AI generates the initial artwork based on the analysis results. The data of this generated artwork is converted into a VR-compatible format.

[0264] Interactive VR space generation

[0265] Terminal: A VR space is constructed based on the artwork data sent from the server, providing a 3D interface that responds to the user's visual and auditory senses.

[0266] Terminal: Captures the actions performed by the user in the VR space (eye tracking, gesture recognition, voice input, emotion recognition using the emotion engine, etc.) in real time and sends the data to the server.

[0267] Real-time reactions and artistic evolution

[0268] Server: The generative AI analyzes the user's reaction and emotional data and uses this data to evolve the artwork, highlighting areas that the user particularly found interesting.

[0269] Server: Data of the evolved artwork is sent to the device, and is instantly reflected in the VR space.

[0270] Device: As new artworks are generated, existing artworks are updated in real time, completely transforming the user experience.

[0271] Feedback accumulation and optimization

[0272] Server: Collects user feedback and ratings and stores them in the generative AI's learning database. This feedback data is used to optimize the next exhibit.

[0273] Terminal: After the exhibition ends, a feedback form will be displayed to the user, providing an interface where they can enter their ratings and comments.

[0274] Further processing for emotion recognition

[0275] On the device: The emotion engine captures emotional data from the user's facial expressions and voice in real time, including analyzing facial expressions from the user's camera footage and tone of voice from the microphone input.

[0276] Server: Analyzes the acquired emotional data and further adjusts the artwork based on the user's emotional state, such as further emphasizing parts that made the user feel surprised or happy.

[0277] Specific examples

[0278] Example 1: When a new user accesses the system and enters their profile information, the generative AI generates an abstract painting in the VR space based on the user's preferences. If the user focuses their gaze on a particular color or shape, and the emotion engine detects the user's joy, the artwork evolves to emphasize that part.

[0279] Example 2: When a repeat user revisits the system, a more personalized artwork is generated based on their past and emotional data, resulting in a more refined and personalized art experience for the user.

[0280] Summary of implementation procedures

[0281] The system combines generative AI, VR technology, eye tracking, gesture recognition, voice input, and emotion recognition to create dynamic and personalized art exhibits that respond to the user's preferences, reactions, and emotions. By utilizing digital space, it can provide new art experiences that transcend physical constraints, and make significant contributions to the fields of education and culture.

[0282] The processing flow will be explained below.

[0283] Step 1:

[0284] User: Accesses the system and logs in.

[0285] Specific operation: The user enters their ID and password and presses the login button.

[0286] Step 2:

[0287] Server: Retrieves the user's profile data.

[0288] Specific operation: Search and extract the user's past art viewing history, preferences, rating data, etc. from the database.

[0289] Step 3:

[0290] Device: Detects when the user is wearing a VR headset and configures the environment.

[0291] Specific operation: Calibrate the VR equipment and obtain the user's body position information.

[0292] Step 4:

[0293] Server: Based on the acquired profile data, the server uses generative AI to analyze the user's art preferences.

[0294] What it does: It feeds profile data into machine learning algorithms to identify user preferences.

[0295] Step 5:

[0296] Server: Generative AI generates an initial artwork based on the analysis results.

[0297] What it does: Generative AI generates digital art by combining appropriate art styles and elements, then converts the data into a VR-compatible format.

[0298] Step 6:

[0299] Server: Sends the generated artwork data to the device.

[0300] Specific operation: Digital art data is compressed and sent to the terminal using a high-speed communication protocol.

[0301] Step 7:

[0302] Terminal: Generates a VR space based on the received art data.

[0303] Specific operation: Using a 3D rendering engine, artwork is displayed in a VR space.

[0304] Step 8:

[0305] User: Experience the artwork in a VR space.

[0306] Specific Actions: Users interact with the artwork using eye tracking, gestures, voice input, and emotion recognition.

[0307] Step 9:

[0308] Device: Sends real-time reaction data from the user's gaze tracking, gesture recognition, voice input, and emotion recognition to the server.

[0309] Specific operation: All data acquired by the gaze tracking sensor, gesture recognition system, microphone, and emotion engine (facial expression analysis, tone of voice analysis) is analyzed and sent to the server in real time.

[0310] Step 10:

[0311] Server: Analyzes user reaction and emotional data, and the generative AI uses this data to evolve the artwork.

[0312] Specific behavior: Based on real-time data, the artwork is modified to add new elements or highlight parts that users like or that evoke an emotional response.

[0313] Step 11:

[0314] Server: The data of the evolved artwork is sent back to the device.

[0315] Specific behavior: Sends updated art data at high speed.

[0316] Step 12:

[0317] Device: Evolved artworks are reflected in the VR space in real time.

[0318] What it does: Instantly update artwork in the VR space and display new content to users.

[0319] Step 13:

[0320] User: After completing the exhibition experience, enter your evaluation in the feedback form.

[0321] Specific action: Fill out the feedback form that appears and provide your rating and comments about the art experience, then submit it.

[0322] Step 14:

[0323] Server: Stores user feedback data and provides it to the generation AI for next display optimization.

[0324] Specific operation: Feedback data is accumulated in a database and used as learning data for the generative AI.

[0325] Example 2

[0326] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0327] Conventional art exhibition systems have difficulty reflecting user preferences, reactions, and emotions in real time, making it difficult to provide a personalized art experience. Furthermore, there is a lack of mechanisms for incorporating user feedback into future exhibitions, making it difficult to provide a continuously optimized experience. To solve these issues, a system is needed that can acquire user profile data, reaction data, and emotional data in real time and evolve artworks based on that data.

[0328] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0329] In this invention, the server includes means for acquiring user attribute information, means for using a generation AI to analyze the user's artistic preferences based on the acquired attribute information, means for rendering artwork generated by the generation AI in a virtual reality space, means for acquiring data based on the user's gaze tracking and motion recognition in real time, means for analyzing the acquired data and using the generation AI to evolve the artwork, means for updating and reflecting the evolved artwork in the virtual reality space, and means for acquiring the user's emotional state in real time and analyzing the data, thereby enabling dynamic and personalized art exhibits based on the user's preferences, reactions, and emotions.

[0330] "User demographic information" is information that indicates a user's personal characteristics and preferences, such as the user's name, age, hobbies, and past activity.

[0331] "Generative AI" is an artificial intelligence technology used to generate or evolve artworks based on user attribute information and reaction data.

[0332] A "virtual reality space" is a three-dimensional digital environment that users access through a VR headset.

[0333] "Eye tracking" is a technology that detects a user's eye movements and gaze position in real time.

[0334] "Motion recognition" is a technology that detects a user's hand movements and body gestures and captures them as data.

[0335] "Real-time acquisition" means that data is collected almost simultaneously and processed immediately.

[0336] "Analysis" is the process of examining acquired data in detail and extracting meaning and patterns.

[0337] "Evolution" is the process of dynamically changing or improving a work of art based on user reactions and emotions.

[0338] "Emotional state" refers to the emotions such as joy, surprise, or anger that a user is feeling at a particular moment.

[0339] An "evaluation form" is an interface that allows users to enter their opinions and thoughts about artworks.

[0340] These definitions clearly describe each element of the system.

[0341] The present invention relates to a system for providing an interactive virtual reality space in which artworks are generated and evolved by recognizing the preferences, reactions, and emotions of users. Specific embodiments will be described below.

[0342] User Awareness and Initial Setup

[0343] server:

[0344] When a user logs in to the system, the server obtains the user's attribute information. This attribute information includes the user's name, age, artistic preferences, past reaction data, etc. For hardware, a general server or cloud infrastructure is used. For specific software, a database management system (e.g., MySQL, PostgreSQL) is used.

[0345] Device:

[0346] It detects when a user puts on a virtual reality (VR) headset and configures the VR environment, including calibrating the VR device (e.g., Oculus Quest 2) and setting up the user's position. The device configures the environment using dedicated VR software (e.g., Unity, Unreal Engine).

[0347] user:

[0348] Users put on a VR headset and controllers to begin accessing the system, and new users are asked to enter basic demographic information such as their name, age, and preferences.

[0349] Analysis of user preferences

[0350] server:

[0351] The acquired attribute information is analyzed and a generative AI is used to identify the user's artistic preferences. In this process, a generative AI model (e.g., GPT-3) is used to identify the user's preferred artistic style (abstract, figurative, color patterns, etc.). Specifically, the following prompt sentences are input to the AI ​​model:

[0352] "Generate abstract paintings that users love and incorporate past preferences."

[0353] Generating the initial artwork

[0354] server:

[0355] Based on the analysis results, the generative AI generates an initial artwork, which is then converted into a VR-compatible format (e.g., GLTF format) and sent to the device.

[0356] Building an interactive VR space

[0357] Device:

[0358] The VR space is constructed based on the artwork data sent from the server. This process uses VR platforms such as Unity and Unreal Engine to provide a three-dimensional interface that responds to the user's visual and auditory senses.

[0359] Real-time data capture and transmission

[0360] Device:

[0361] The system captures user actions (eye tracking, motion recognition, and voice input) in real time and sends the data to a server. For eye tracking, it uses a built-in camera and sensors, and for motion recognition, it uses a remote controller and hand tracking technology. For voice input, it uses a microphone, which is then analyzed by voice recognition software (e.g., Google Cloud Speech-to-Text).

[0362] Device: Using an emotion engine (e.g., Emotion API), emotional data is acquired from the user's facial expressions and voice and sent to the server. Specifically, facial expressions are analyzed from the user's camera footage, and the tone of voice is analyzed from microphone input.

[0363] Real-time reactions and artistic evolution

[0364] server:

[0365] The AI ​​analyzes user reaction and emotional data and uses this data to evolve the artwork. For example, if a user expresses interest in a particular color or shape, it will emphasize that part.

[0366] server:

[0367] By sending data of the evolved artwork to the device, it is instantly reflected in the VR space.

[0368] Device:

[0369] As new artworks are generated, existing works are updated in real time, providing a fresh user experience.

[0370] Feedback collection and optimization

[0371] server:

[0372] User feedback and ratings are collected and stored in the generative AI's learning database, which will further optimize the next art exhibit.

[0373] Device:

[0374] After the exhibition ends, a feedback form will be displayed to users, providing an interface where they can enter their ratings and comments.

[0375] Further processing for emotion recognition

[0376] Device:

[0377] The emotion engine captures emotional data from the user's facial expressions and voice in real time, including analyzing facial expressions from the user's camera footage and tone of voice from microphone input.

[0378] server:

[0379] The acquired emotional data is analyzed and the artwork is further adjusted based on the user's emotional state, for example by further emphasizing parts that made the user feel surprised or happy.

[0380] This system will enable personalized art exhibitions based on the user's preferences, reactions, and emotions, and will also provide a new art experience that transcends physical constraints through the use of digital technology.

[0381] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0382] Step 1:

[0383] When a user logs into the system, the server obtains the user's attribute information (name, age, artistic preferences, and past response data).

[0384] Input: User ID and login information

[0385] Data processing: Search the database using the user ID as a key and obtain the corresponding user information

[0386] Output: Retrieved user attribute information

[0387] Step 2:

[0388] The server analyzes the acquired attribute information and uses generative AI to identify the user's artistic preferences.

[0389] Input: User attribute information

[0390] Data calculation: Input prompts into the generative AI model and perform analysis that reflects user preferences.

[0391] Output: Information about the user's artistic preferences

[0392] Example prompt: “Generate abstract paintings that users will love and incorporate their past preferences.”

[0393] Step 3:

[0394] The server uses generative AI to generate an initial artwork based on the user's preferences.

[0395] Input: Information about the user's artistic preferences

[0396] Data Computation: Generative AI model generates initial artwork based on prompt text

[0397] Output: Initial artwork data (converted to VR format)

[0398] Operation: The generated artwork data is converted into VR-compatible GLTF format and sent to the device.

[0399] Step 4:

[0400] The device creates a VR space based on the artwork data sent from the server.

[0401] Input: Initial artwork data in GLTF format

[0402] Data processing: Using VR software such as Unity or Unreal Engine, we create a three-dimensional VR space.

[0403] Output: The VR space experienced by the user

[0404] Action: Calibrate the VR device and set up the device to understand the user's position.

[0405] Step 5:

[0406] The device captures the user's gaze tracking, movement recognition, and voice input in real time within the VR space and sends the data to a server.

[0407] Input: User gaze, movement, and voice data

[0408] Data processing: Capture and process data using eye-tracking sensors, gesture recognition sensors, and voice recognition software

[0409] Output: Captured real-time data

[0410] How it works: Uses the emotion engine to get user emotion data and send it to the server

[0411] Step 6:

[0412] The server analyzes the real-time data it receives, and the generative AI uses this data to evolve the artwork.

[0413] Input: Captured real-time data and sentiment data

[0414] Data Computation: Generative AI models analyze real-time data and dynamically update artwork based on user responses

[0415] Output: Improved artwork data

[0416] How it works: If a particular color or shape shows interest, it highlights that area in near real time.

[0417] Step 7:

[0418] The server transmits data of the evolved artwork to the terminal.

[0419] Input: Evolved artwork data

[0420] Data processing: Reconverting the evolved artwork data into VR format

[0421] Output: Reconverted artwork data

[0422] Operation: Transmitted to the device and reflected instantly in the VR space

[0423] Step 8:

[0424] The device updates the existing VR space in real time as new artworks are created.

[0425] Input: Evolved artwork data

[0426] Data processing: Using VR software such as Unity or Unreal Engine, we update existing VR spaces with new artwork.

[0427] Output: Updated VR space

[0428] What it does: A completely new user experience

[0429] Step 9:

[0430] The server collects user feedback and ratings and stores them in the generative AI's learning database.

[0431] Input: User feedback and rating data

[0432] Data processing: Data collected through the feedback form is saved in the AI ​​learning database.

[0433] Output: Accumulated feedback data

[0434] Action: The display will be further optimized from next time onwards.

[0435] Step 10:

[0436] After the exhibition ends, the terminal displays a feedback form to users, providing an interface where they can enter their ratings and comments.

[0437] Input: User ratings and comments

[0438] Data processing: Evaluation data is collected using a feedback form and sent to the server.

[0439] Output: Collected assessment data

[0440] Behavior: Updates the interface presented to the user to prompt for feedback

[0441] In this way, by clarifying the specific actions, inputs and outputs performed at each step, an interactive VR artwork is provided that evolves in real time in response to the user's preferences, reactions and emotions.

[0442] (Application example 2)

[0443] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0444] Conventional art exhibition systems have struggled to generate personalized artworks that reflect users' preferences, reactions, and emotions in real time, and lacked a system for optimizing the next exhibition using feedback and emotional data, making it difficult to increase user satisfaction.

[0445] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0446] In this invention, the server includes means for acquiring user profile data, means for using a generation AI to analyze the user's art preferences based on the acquired profile data, means for rendering artwork generated by the generation AI in a virtual reality space, means for acquiring the user's gaze tracking, gesture recognition, voice input, and emotional data in real time, means for analyzing the acquired data and using the generation AI to evolve the artwork, means for updating and reflecting the evolved artwork in the virtual reality space, means for enhancing or adjusting the artwork based on the user's emotional data, and means for collecting user feedback and accumulating it in the generation AI's learning database, thereby enabling the provision of personalized artwork that reflects the user's preferences, reactions, and emotions in real time.

[0447] "User Profile Data" means personal information such as your name, age, art preferences, and past interaction data.

[0448] "Generative AI" is artificial intelligence that uses machine learning algorithms to generate and evolve artworks based on user data.

[0449] A "virtual reality space" is a three-dimensional interface space created using virtual reality technology that users can experience visually and aurally.

[0450] "Eye tracking" is a technology that captures and analyzes a user's eye movements in real time.

[0451] "Gesture recognition" is a technology that captures a user's hand and body movements and analyzes those movements.

[0452] "Voice input" is a technology that acquires and analyzes the user's voice data.

[0453] "Emotional data" is data that indicates the emotional state of a user, obtained from the user's facial expression, tone of voice, etc.

[0454] An "evolved artwork" is a work that has undergone some changes and adjustments to the original artwork based on user reaction and emotional data.

[0455] "Feedback" refers to information such as ratings and comments that users enter about artworks.

[0456] The "learning database" is a database that accumulates past data and feedback that the generative AI will use the next time it generates an artwork.

[0457] "Real time" is a time concept that refers to a system's immediate response and processing without delay.

[0458] This invention provides a system for providing an interactive virtual reality space in which artwork is generated and evolved based on the user's preferences, reactions, and emotions. Specific embodiments of this system are described below.

[0459] When a user logs into the system, the server retrieves profile data. This profile data includes the user's name, age, art preferences, and past reaction data. The device detects whether the user is wearing a smartphone or head-mounted display (HMD) and configures the environment. This includes calibrating the VR equipment and configuring the user's position in the environment. The user is required to enter basic profile information when accessing the system for the first time.

[0460] The server analyzes the acquired profile data and uses a generative AI to analyze the user's art preferences. This process uses machine learning algorithms to identify trends in the user's preferred art styles (abstract, figurative, color patterns, etc.). The generative AI generates an initial artwork based on the analysis results, and the data for this generated artwork is converted into a virtual reality-compatible format.

[0461] The device creates a virtual reality space based on the artwork data sent from the server, providing a three-dimensional interface that responds to the user's visual and auditory senses. Furthermore, the device captures the user's actions in the virtual reality space (eye tracking, gesture recognition, voice input, emotional data acquisition, etc.) in real time and sends the data to the server.

[0462] The server analyzes the user's reaction and emotional data. Based on the analyzed data, the generation AI evolves the artwork. Specifically, it makes changes such as emphasizing parts that the user particularly finds interesting. The data for the evolved artwork is then sent back to the device and instantly reflected in the virtual reality space. In this way, when a new artwork is generated, the existing artwork is updated in real time, completely renewing the user's experience.

[0463] The server collects user feedback and ratings and stores them in the generative AI's learning database. This feedback data is used to optimize the next exhibition. After the exhibition ends, the terminal displays a feedback form to the user, providing an interface where they can enter ratings and comments. The emotion engine obtains emotional data from the user's facial expressions and voice in real time. This includes analyzing facial expressions from the user's camera footage and voice tone from microphone input, and analyzes the obtained emotional data to further adjust the artwork based on their emotional state.

[0464] As a concrete example, consider a scenario where a user generates an artwork using a prompt such as, "Create a colorful abstract painting. In the past, the user has preferred blue and warm color patterns." The artwork generated based on this prompt evolves according to the user's response, for example, emphasizing the area where the user's gaze is directed or changing colors to reflect emotions. As a result, the user can enjoy a more personalized art experience.

[0465] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0466] Step 1:

[0467] A user logs into the system and enters profile data.

[0468] Input: User's name, age, art preferences, and past response data.

[0469] Output: User profile data.

[0470] Specific operation: A user accesses the system using a smartphone or head-mounted display (HMD) and enters profile information when logging in for the first time.

[0471] Step 2:

[0472] The server retrieves the user's profile data and uses generative AI to analyze the user's art preferences.

[0473] Input: The user's profile data obtained in step 1.

[0474] Output: Analysis of user's art preferences.

[0475] What it does: The server analyzes your profile data and uses machine learning algorithms to identify trends in your preferred art styles (abstract, figurative, color patterns, etc.).

[0476] Step 3:

[0477] Generative AI generates an initial artwork based on the analysis and converts it into a VR-compatible format.

[0478] Input: The analysis results from step 2.

[0479] Output: Virtual reality compatible artwork data.

[0480] Specific operation: The generation AI generates a prompt (e.g., "Generate a colorful abstract painting. Users have previously preferred blue and warm color patterns") and generates an artwork based on it. The generated data is converted into a VR format.

[0481] Step 4:

[0482] The device creates a virtual reality space based on the data of the artwork sent from the server.

[0483] Input: Virtual reality-ready artwork data generated in step 3.

[0484] Output: A work of art displayed in a virtual reality space.

[0485] How it works: When a user wears a head-mounted display (HMD), the device displays the artwork, providing the user with a three-dimensional interface that supports both visual and auditory senses.

[0486] Step 5:

[0487] The device captures the user's gaze tracking, gesture recognition, voice input, and emotional data in real time and transmits it to the server.

[0488] Input: User gaze, gesture, voice, and emotion data.

[0489] Output: User response data captured in real time.

[0490] How it works: The device uses the camera and microphone to analyze the user's facial expressions and tone of voice, capture eye tracking and gesture recognition data, and transmit this data to a server in real time.

[0491] Step 6:

[0492] The server analyzes user reaction data and uses generative AI to evolve the artwork.

[0493] Input: User response data obtained in step 5.

[0494] Output: Evolved artwork data.

[0495] How it works: The server analyzes the areas of interest and changes in emotions of the user, and the generative AI modifies the artwork based on that data, for example by emphasizing certain colors or shapes.

[0496] Step 7:

[0497] The device updates and reflects the evolved artwork in the virtual reality space.

[0498] Input: The evolved artwork data generated in step 6.

[0499] Output: An updated artwork in a virtual reality space.

[0500] What it does: The device updates the artwork already displayed in real time, providing the user with a new art experience.

[0501] Step 8:

[0502] The server collects user feedback and ratings and stores them in the generative AI's learning database.

[0503] Input: User feedback and ratings.

[0504] Output: Feedback data stored in the training database.

[0505] Specific operation: After the exhibition ends, the terminal displays a feedback form to the user. The user's ratings and comments are collected, and the server provides this to the generation AI as learning data.

[0506] Step 9:

[0507] The terminal obtains the user's emotional data in real time and further adjusts the artwork based on the user's emotional state.

[0508] Input: User facial and vocal emotion data.

[0509] Output: A tailored piece of art.

[0510] How it works: The device analyzes the user's emotions based on data obtained from camera footage and microphone input, and then enhances or modifies the artwork based on that.

[0511] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0512] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0513] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0514] [Second embodiment]

[0515] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0516] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0517] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0518] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0519] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0520] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0521] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0522] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0523] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0524] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0525] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0526] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0527] This invention relates to a system for providing an interactive virtual reality (VR) space in which artwork is generated and evolved according to the user's preferences and reactions. Specific aspects of the system are described below.

[0528] User Awareness and Initial Setup

[0529] Server: When a user logs into the system, the server retrieves the user's profile data, which includes the user's name, age, art preferences, past reaction data, etc.

[0530] Device: Detects when the user puts on a VR headset and prepares the VR environment, including calibrating the VR equipment and configuring it to understand the user's position in the environment.

[0531] User: A user puts on a VR headset and controllers and begins accessing the system. First-time users are prompted to enter basic profile information.

[0532] Analysis of user preferences

[0533] Server: Analyzes user profile data and uses generative AI to analyze the user's art preferences. This process uses machine learning algorithms to identify trends in the user's preferred art styles (abstract, figurative, color patterns, etc.).

[0534] Server: The generative AI generates the initial artwork based on the analysis results. The data of this generated artwork is converted into a VR-compatible format.

[0535] Interactive VR space generation

[0536] Terminal: A VR space is constructed based on the artwork data sent from the server, providing a 3D interface that responds to the user's visual and auditory senses.

[0537] Device: Captures the user's actions in the VR space (eye tracking, gesture recognition, voice input, etc.) in real time and sends the data to the server.

[0538] Real-time reactions and artistic evolution

[0539] Server: The generative AI analyzes user reaction data and uses it to evolve the artwork, highlighting areas that the user particularly found interesting.

[0540] Server: Data of the evolved artwork is sent to the device, and is instantly reflected in the VR space.

[0541] Device: As new artworks are generated, existing artworks are updated in real time, completely transforming the user experience.

[0542] Feedback accumulation and optimization

[0543] Server: Collects user feedback and ratings and stores them in the generative AI's learning database. This feedback data is used to optimize the next exhibit.

[0544] Terminal: After the exhibition ends, a feedback form will be displayed to the user, providing an interface where they can enter their ratings and comments.

[0545] Specific examples

[0546] Example 1: When a new user accesses the system and enters their profile information, the generative AI generates an abstract painting in the VR space based on the user's preferences. If the user focuses their gaze on a particular color or shape, the artwork evolves to emphasize that aspect.

[0547] Example 2: When a repeat user revisits the system, artworks are generated that are more tailored to the user based on their past data, resulting in a more refined and personalized art experience for the user.

[0548] As described above, this system combines generative AI and VR technology to create dynamic art exhibits that respond to the user's preferences and reactions. By utilizing digital space, it can provide new art experiences that transcend physical constraints, and can also make a significant contribution to the fields of education and culture.

[0549] The processing flow will be explained below.

[0550] Step 1:

[0551] User: Accesses and logs into the system.

[0552] Specific operation: The user enters their ID and password and presses the login button.

[0553] Step 2:

[0554] Server: Retrieves the user's profile data.

[0555] Specific operation: Search and extract the user's past art viewing history, preferences, rating data, etc. from the database.

[0556] Step 3:

[0557] Device: Detects when the user is wearing a VR headset and configures the environment.

[0558] Specific operation: Calibrate the VR equipment and obtain the user's body position information.

[0559] Step 4:

[0560] Server: Based on the acquired profile data, the server uses generative AI to analyze the user's artistic preferences.

[0561] What it does: It feeds profile data into machine learning algorithms to identify user preferences.

[0562] Step 5:

[0563] Server: Generative AI generates an initial artwork based on the analysis results.

[0564] What it does: The AI ​​model generates digital art by combining appropriate art styles and elements, then converts the data into a VR-compatible format.

[0565] Step 6:

[0566] Server: Sends the generated artwork data to the device.

[0567] Specific operation: Digital art data is compressed and sent to the terminal using a high-speed communication protocol.

[0568] Step 7:

[0569] Terminal: Generates a VR space based on the received art data.

[0570] Specific operation: Using a 3D rendering engine, artwork is displayed in a VR space.

[0571] Step 8:

[0572] User: Experience the artwork in a VR space.

[0573] Specific actions: Users interact with the artwork using eye tracking and gestures.

[0574] Step 9:

[0575] Device: Sends real-time reaction data from the user's gaze tracking and gesture recognition to the server.

[0576] Specific operation: Data acquired by the gaze tracking sensor and gesture recognition system is analyzed and sent to the server in real time.

[0577] Step 10:

[0578] Server: Analyzes user reaction data, and the generative AI uses this data to evolve the artwork.

[0579] Specific behavior: Based on real-time data, the system makes changes to the artwork, such as adding new elements or emphasizing parts that users like.

[0580] Step 11:

[0581] Server: The data of the evolved artwork is sent back to the device.

[0582] Specific behavior: Sends updated art data at high speed.

[0583] Step 12:

[0584] Device: Evolved artworks are reflected in the VR space in real time.

[0585] What it does: Instantly update artwork in the VR space and display new content to users.

[0586] Step 13:

[0587] User: After completing the exhibition experience, enter your evaluation in the feedback form.

[0588] Specific action: Fill out the feedback form that appears and provide your rating and comments about the art experience, then submit it.

[0589] Step 14:

[0590] Server: Stores user feedback data and provides it to the generation AI for next display optimization.

[0591] Specific operation: Feedback data is accumulated in a database and used as learning data for the generative AI.

[0592] Example 1

[0593] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0594] Conventional art exhibition systems have struggled to dynamically generate and evolve artworks based on individual user preferences and real-time responses. They also have limited means to effectively incorporate interactive data, such as user gaze tracking and gesture recognition. As a result, the experience for each user is not fully personalized, and there is no appropriate mechanism for incorporating user feedback into future exhibitions.

[0595] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0596] In this invention, the server includes means for acquiring user profile data, means for using artificial intelligence to analyze user preferences based on the acquired user profile data, means for rendering a digital artwork generated by the artificial intelligence in a virtual reality space, means for acquiring data based on the user's gaze tracking and motion recognition in real time, means for analyzing the acquired data and using artificial intelligence to evolve the digital artwork, and means for updating and reflecting the evolved digital artwork in the virtual reality space, thereby enabling a dynamic and personalized art experience that reflects individual user preferences and real-time reactions.

[0597] "User profile data" refers to any information related to a user who logs into the system, such as the user's name, age, art preferences, and past response data.

[0598] "Artificial intelligence" refers to technologies including machine learning algorithms and deep learning models that analyze user profile data and real-time reaction data to generate and evolve appropriate artworks.

[0599] "Digital artwork" refers to art generated by artificial intelligence, the data of which is converted into a format (e.g., FBX or OBJ) that can be rendered in a virtual reality space.

[0600] "Virtual reality space" refers to a three-dimensional digital environment that users can immerse themselves in using devices such as VR headsets.

[0601] "Eye tracking" refers to a technology that tracks the direction a user is looking or a specific object in a virtual reality space in real time.

[0602] "Motion recognition" refers to the technology of capturing a user's gestures and body movements with sensors and analyzing them.

[0603] "Real-time data" refers to data obtained through user gaze tracking and / or motion recognition that is processed and analyzed in real time.

[0604] MODE FOR CARRYING OUT THE INVENTION

[0605] This invention relates to a system that provides an interactive virtual reality (VR) space in which digital artwork is generated and evolved according to the preferences and reactions of users. Specific embodiments will be described below.

[0606] User Awareness and Initial Setup

[0607] Server: When a user logs into the system, the server retrieves the user's profile data (such as name, age, art preferences, past reaction data, etc.) This profile data is retrieved using database queries, and the data is cleansed and analyzed using Python libraries (e.g., Pandas, NumPy).

[0608] Device: When a user puts on a VR headset (e.g., a general-purpose VR headset), the device detects this information and prepares the VR environment, including calibrating the VR device and configuring it to understand the user's position (e.g., 6DOF tracking).

[0609] User: The user puts on the VR headset and controllers and begins accessing the system. First-time users are asked to enter basic profile information (such as name, age, and art preferences) through a form in the interface.

[0610] Analysis of user preferences

[0611] Server: The server analyzes the acquired user profile data and uses generative AI to analyze the user's art preferences. In this process, machine learning algorithms (e.g., TensorFlow, PyTorch) are used to identify trends in the user's preferred art styles (abstract art, figurative art, color patterns, etc.).

[0612] Server: Generative AI (e.g., GPT-3, GANs) generates the initial digital artwork based on the analysis results. The Assimp library is used to convert this generated artwork data into a VR-compatible format (e.g., FBX format or OBJ format).

[0613] Interactive VR space generation

[0614] Terminal: Using Unity or Unreal Engine, a VR space is created based on the artwork data sent from the server. This provides a 3D interface that responds to the user's vision and hearing.

[0615] Device: Configure it using the necessary APIs (e.g., OpenVR SDK, general-purpose VR SDK) to capture the user's actions in the VR space (e.g., eye tracking, motion recognition, voice input, etc.) in real time.

[0616] Real-time reactions and artistic evolution

[0617] Server: Analyzes the user's real-time reaction data (eye tracking data, motion recognition data, etc.), and the generative AI evolves the artwork based on this. Specifically, computer vision technology (e.g., OpenCV) is used to analyze the gaze data.

[0618] Server: Sends the evolved artwork data to the device. This process uses WebSocket to exchange data in real time.

[0619] Device: When a new artwork is generated, the existing artwork is instantly updated in the VR space, leveraging the real-time rendering capabilities of Unity and Unreal Engine, so the changes are instantly reflected in the user's vision.

[0620] Feedback accumulation and optimization

[0621] Server: After the exhibition, the feedback and evaluations collected from users are stored in the AI's learning database. This data will be used for future art generation.

[0622] Terminal: After the exhibition ends, a feedback form is displayed to the user, providing an interface where they can enter their ratings and comments. This form data is collected using HTML and JavaScript and sent to the server.

[0623] Specific examples

[0624] Example 1:

[0625] Server: When a new user logs in, it collects profile information and queries the database. Python is used to clean the data and extract preferences. The generative AI generates abstract art based on the data and converts it into FBX format.

[0626] Device: Receives artwork data and displays it in the VR space using Unity. When the user directs their gaze at a specific color or shape, eye-tracking data is sent to the server.

[0627] Server: To evolve the artwork based on the eye-tracking data, the generative AI updates the artwork design. It converts the new data into a VR format and sends it to the device.

[0628] Device: The new artwork is instantly updated in the VR space, allowing users to experience the changes in real time.

[0629] Example 2:

[0630] Server: When a repeat user accesses the site again, the initial settings are made based on the user's past profile data and responses, and an art piece is generated that is further tailored to the user.

[0631] Device: Compares existing art with new art and dynamically updates the VR space. When the user responds with gestures to specific shapes, the information is sent to the server in real time.

[0632] Server: The artwork is partially evolved based on gesture recognition data and the information is sent to the device.

[0633] On your device: Display updated artwork so that your art experience is more refined and personalized than the last time.

[0634] Prompt Sentence Examples

[0635] "Describe the process of generating abstract paintings based on user profile information and analyzing the user's eye-tracking data to evolve the artwork."

[0636] As described above, this system combines generative AI and VR technology to provide a dynamic art experience that responds to the user's preferences and reactions. By utilizing digital space, it can provide new art experiences that transcend physical constraints and make significant contributions to the fields of education and culture.

[0637] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0638] Program processing flow

[0639] Step 1: User Awareness and Initial Setup

[0640] Server: When a user logs in to the system, the server retrieves the user's profile data (such as name, age, art preferences, and past reaction data) using a database query and cleanses the data using Python libraries (e.g., Pandas and NumPy).

[0641] Input: User login information.

[0642] Output: Cleansed user profile data.

[0643] Device: When the user puts on the VR headset, the device detects this information and prepares the VR environment. Specifically, it calibrates the VR equipment and sets up the user's position (for example, 6DOF tracking).

[0644] Input: VR headset wearing information.

[0645] Output: Calibrated VR environment setup.

[0646] User: The user puts on the VR headset and controllers and begins accessing the system. First-time users enter basic profile information (such as name, age, and art preferences) through a form in the interface.

[0647] Input: Basic profile information.

[0648] Output: The entered profile information.

[0649] Step 2: Analyzing user preferences

[0650] Server: The server analyzes user profile data and uses generative AI (e.g., TensorFlow or PyTorch) to analyze the user's art preferences. It uses machine learning algorithms to identify trends in the user's preferred art styles (e.g., abstract, figurative, color patterns, etc.).

[0651] Input: Cleansed user profile data.

[0652] Output: User's art preference information.

[0653] Server: The generative AI generates the initial digital artwork based on the analysis results. The generated artwork data is converted into a VR-compatible format (e.g., FBX or OBJ format) using the Assimp library.

[0654] Input: User's art preference information.

[0655] Output: Digital artwork data in a VR-compatible format.

[0656] Step 3: Creating an interactive VR space

[0657] Device: Using Unity or Unreal Engine, a VR space is created based on the artwork data sent from the server, providing a 3D interface that responds to the user's visual and auditory senses.

[0658] Input: Digital artwork data in a VR-compatible format.

[0659] Output: The constructed VR space.

[0660] Device: Captures the user's actions in the VR space (eye tracking, motion recognition, voice input, etc.) in real time and sends the data to the server. Uses the required API (e.g., OpenVR SDK, general-purpose VR SDK).

[0661] Input: User operation data.

[0662] Output: Operation data sent to the server.

[0663] Step 4: Real-time reactions and artistic evolution

[0664] Server: Analyzes the user's real-time reaction data (eye tracking data, motion recognition data, etc.), and the generative AI evolves the artwork based on this data. Computer vision technology (e.g., OpenCV) is used to analyze the gaze data.

[0665] Input: Real-time user response data.

[0666] Output: Evolved digital artwork data.

[0667] Server: Sends the evolved artwork data to the device. Data is exchanged in real time using WebSocket.

[0668] Input: Evolved digital artwork data.

[0669] Output: Artwork data sent to the device.

[0670] Device: As new artworks are generated, existing artworks are updated in real time within the VR space, utilizing the real-time rendering capabilities of Unity and Unreal Engine.

[0671] Input: Evolved digital artwork data.

[0672] Output: Updated VR space.

[0673] Step 5: Gather feedback and optimize

[0674] Server: After the exhibition is over, the feedback and evaluations collected from users are stored in the Generative AI's learning database. This data is used to optimize the next exhibition.

[0675] Input: User feedback data.

[0676] Output: Saved feedback data.

[0677] Terminal: After the exhibition ends, a feedback form is displayed to the user, providing an interface where they can enter their ratings and comments. The form data is collected using HTML and JavaScript and sent to the server.

[0678] Input: User ratings and comments.

[0679] Output: The feedback form data sent to the server.

[0680] Through these processing steps, the system provides a dynamic art experience that responds to user preferences and real-time responses.

[0681] (Application example 1)

[0682] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0683] Conventional entertainment systems for autonomous vehicles have the drawback of being difficult to provide content that responds to individual user preferences and real-time responses, and are difficult to provide an experience with enhanced visual and tactile interactivity due to the limited environment inside the vehicle.

[0684] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0685] In this invention, the server includes means for acquiring user profile data, means for using a generation AI to analyze the user's art preferences based on the acquired profile data, means for rendering artwork generated by the generation AI in a virtual reality space, means for acquiring data based on the user's gaze tracking and gesture recognition in real time, means for analyzing the acquired data and using the generation AI to evolve the artwork, means for updating and reflecting the evolved artwork in the virtual reality space, and means for providing a terminal for using these means within an autonomous vehicle, thereby enabling the provision of a high-quality, personalized art experience in real time even within an autonomous vehicle.

[0686] "User Profile Data" means data that includes personal information about you, such as your name, age, and artistic preferences.

[0687] "Generative AI" is artificial intelligence that uses machine learning algorithms to generate and evolve artworks based on user preferences.

[0688] A "virtual reality space" is a three-dimensional virtual space that provides users with an immersive experience through visual and auditory interfaces.

[0689] "Eye tracking" is a technology that detects and analyzes the user's eye movements in real time.

[0690] "Gesture recognition" is a technology that uses cameras and sensors to detect and analyze the movements of a user's hands and fingers.

[0691] "Real-time acquisition means" refers to a system or device that instantly collects interaction data such as user gaze and gestures.

[0692] An "autonomous vehicle" is a vehicle that drives autonomously using artificial intelligence and sensor technology.

[0693] A "terminal" is a device or system (e.g., a head-mounted display) that allows a user to access a virtual reality space.

[0694] A "feedback form" is an interface that allows users to enter their opinions and evaluations of artworks.

[0695] The "generative AI learning database" is a database used to accumulate user feedback and reaction data and improve the performance of the generative AI.

[0696] The embodiments of the present invention will be specifically described below.

[0697] Retrieving a user profile

[0698] When a user logs into the system, the server retrieves the user's profile data, which includes the user's name, age, art preferences, past interaction data, etc. This provides the basis for collecting data to provide a personalized experience for each user.

[0699] User Awareness and Initial Setup

[0700] The device detects when the user is wearing a VR headset and prepares the VR environment, including calibrating the head-mounted display (e.g., Oculus Quest) and configuring it to understand the user's position. When a user first accesses the system, they are required to enter basic profile information.

[0701] Art taste analysis

[0702] The server analyzes the profile data and uses generative AI to analyze the user's art preferences. This process uses machine learning algorithms to identify trends in the user's preferred art styles (abstract, figurative, color patterns, etc.).

[0703] Generating and Rendering Initial Artwork

[0704] The generative AI generates an initial artwork based on the user's preferences. The data for this artwork is converted into a format compatible with the virtual reality space and sent to the device. The device then creates a virtual reality space based on the data sent from the server. This provides a 3D interface that responds to the user's visual and auditory senses.

[0705] Interactive operation and real-time updates

[0706] The device captures the user's actions in the virtual reality space (eye tracking, gesture recognition, voice input, etc.) in real time and sends the data to the server. The server analyzes the user's reaction data, and the generative AI uses this data to evolve the artwork. Changes are made, such as emphasizing areas that the user has shown particular interest in. Data on the evolved artwork is sent to the device and instantly reflected in the virtual reality space, completely transforming the user's experience.

[0707] Feedback collection and optimization

[0708] The server collects user feedback and ratings and stores them in the generative AI's learning database. This feedback data is used to optimize the next exhibit. After the exhibit is over, the terminal displays a feedback form to the user, providing an interface where they can enter ratings and comments.

[0709] Use in self-driving vehicles

[0710] This system is installed in autonomous vehicles, allowing users to enjoy art experiences using VR headsets while traveling. The in-car device captures the user's actions in real time and works with a server to evolve the artwork, providing immersive entertainment even while traveling.

[0711] Examples:

[0712] Example 1: A new user accesses the system and enters their profile information. Generative AI generates an abstract painting in a virtual reality space based on the user's preferences. If the user focuses their gaze on a particular color or shape, the artwork evolves to emphasize that feature.

[0713] Example 2: When a returning user revisits the system, the system generates more personalized artwork based on their past data, resulting in a more refined and personalized art experience.

[0714] Example prompt sentence:

[0715] "Develop an app for autonomous vehicles that allows passengers to use VR headsets to experience and evolve art according to their preferences."

[0716] In this way, this invention combines generative AI and virtual reality technology to create dynamic art exhibits that respond to user preferences and reactions, providing a new entertainment experience even inside autonomous vehicles.

[0717] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0718] Step 1:

[0719] When a user logs in to the system, the server retrieves the user's profile data. The profile data includes the user's name, age, art preferences, and past response data. This provides the basis for the user to receive a personalized experience. The input is the user's login information, and the output is the user's profile data.

[0720] Step 2:

[0721] The device detects when the user puts on a VR headset and prepares the virtual reality environment. This includes calibrating the head-mounted display (e.g., Oculus Quest) and determining the user's position. The input is the state of the VR headset and user position information, and the output is information that the virtual reality environment is ready.

[0722] Step 3:

[0723] The server analyzes the acquired profile data and uses generative AI to analyze the user's art preferences. Machine learning algorithms are used to identify trends in the user's preferred art styles (abstract, figurative, color patterns, etc.). The input is the profile data, and the output is the analysis of the user's art preferences.

[0724] Step 4:

[0725] The server uses a generative AI to generate an initial artwork based on the user's preferences, converts it into a format compatible with the virtual reality space, and sends it to the device. The input is the user's art preference analysis results, and the output is VR-compatible initial artwork data.

[0726] Step 5:

[0727] The device constructs a virtual reality space based on the artwork data received from the server, providing a 3D interface that responds to the user's visual and auditory senses. The input is the initial artwork data, and the output is the constructed virtual reality space.

[0728] Step 6:

[0729] The device captures the user's actions in the virtual reality space (eye tracking, gesture recognition, voice input) in real time and sends the data to the server. The input is the user's action data, and the output is the data sent to the server.

[0730] Step 7:

[0731] The server analyzes the user's operation data, and the generative AI evolves the artwork based on this data. In particular, it makes changes such as emphasizing the parts that the user showed interest in. The input is the user's operation data, and the output is the evolved artwork data.

[0732] Step 8:

[0733] The server sends the evolved artwork data to the device, which then instantly reflects it in the virtual reality space. The input is the evolved artwork data, and the output is the updated virtual reality space.

[0734] Step 9:

[0735] The server collects user feedback and ratings and stores them in the generative AI's learning database. This feedback data is used to optimize the next display. The input is user feedback, and the output is an updated learning database.

[0736] Step 10:

[0737] The terminal displays a feedback form to the user after the exhibition has ended, providing an interface where users can enter their ratings and comments. The input triggers the end of the exhibition, and the output is the feedback information from the user.

[0738] The entire process enables a personalized art experience to be delivered in real time within the autonomous vehicle, and the system dynamically adapts to the user's preferences and reactions, delivering high-quality, immersive entertainment.

[0739] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0740] This invention relates to a system for providing an interactive virtual reality (VR) space that generates and evolves artwork by recognizing the user's preferences, reactions, and emotions. Specific aspects of the system are described below.

[0741] User Awareness and Initial Setup

[0742] Server: When a user logs into the system, the server retrieves the user's profile data, which includes the user's name, age, art preferences, past reaction data, etc.

[0743] Device: Detects when the user puts on a VR headset and configures the environment, including calibrating the VR equipment and locating the user in the environment.

[0744] User: A user puts on a VR headset and controllers and begins accessing the system. First-time users are prompted to enter basic profile information.

[0745] Analysis of user preferences

[0746] Server: Analyzes user profile data and uses generative AI to analyze the user's art preferences. This process uses machine learning algorithms to identify trends in the user's preferred art styles (abstract, figurative, color patterns, etc.).

[0747] Server: The generative AI generates the initial artwork based on the analysis results. The data of this generated artwork is converted into a VR-compatible format.

[0748] Interactive VR space generation

[0749] Terminal: A VR space is constructed based on the artwork data sent from the server, providing a 3D interface that responds to the user's visual and auditory senses.

[0750] Terminal: Captures the user's actions in the VR space (eye tracking, gesture recognition, voice input, emotion recognition using the emotion engine, etc.) in real time and sends the data to the server.

[0751] Real-time reactions and artistic evolution

[0752] Server: The generative AI analyzes the user's reaction and emotional data and uses this data to evolve the artwork, highlighting areas that the user particularly found interesting.

[0753] Server: Data of the evolved artwork is sent to the device, and is instantly reflected in the VR space.

[0754] Device: As new artworks are generated, existing artworks are updated in real time, completely transforming the user experience.

[0755] Feedback accumulation and optimization

[0756] Server: Collects user feedback and ratings and stores them in the generative AI's learning database. This feedback data is used to optimize the next exhibit.

[0757] Terminal: After the exhibition ends, a feedback form will be displayed to the user, providing an interface where they can enter their ratings and comments.

[0758] Further processing for emotion recognition

[0759] On the device: The emotion engine captures emotional data from the user's facial expressions and voice in real time, including analyzing facial expressions from the user's camera footage and tone of voice from the microphone input.

[0760] Server: Analyzes the acquired emotional data and further adjusts the artwork based on the user's emotional state, such as further emphasizing parts that made the user feel surprised or happy.

[0761] Specific examples

[0762] Example 1: When a new user accesses the system and enters their profile information, the generative AI generates an abstract painting in the VR space based on the user's preferences. If the user focuses their gaze on a particular color or shape, and the emotion engine detects the user's joy, the artwork evolves to emphasize that part.

[0763] Example 2: When a repeat user revisits the system, a more personalized artwork is generated based on their past and emotional data, resulting in a more refined and personalized art experience for the user.

[0764] Summary of implementation procedures

[0765] The system combines generative AI, VR technology, eye tracking, gesture recognition, voice input, and emotion recognition to create dynamic and personalized art exhibits that respond to the user's preferences, reactions, and emotions. By utilizing digital space, it can provide new art experiences that transcend physical constraints, and make significant contributions to the fields of education and culture.

[0766] The processing flow will be explained below.

[0767] Step 1:

[0768] User: Accesses and logs into the system.

[0769] Specific operation: The user enters their ID and password and presses the login button.

[0770] Step 2:

[0771] Server: Retrieves the user's profile data.

[0772] Specific operation: Search and extract the user's past art viewing history, preferences, rating data, etc. from the database.

[0773] Step 3:

[0774] Device: Detects when the user is wearing a VR headset and configures the environment.

[0775] Specific operation: Calibrate the VR equipment and obtain the user's body position information.

[0776] Step 4:

[0777] Server: Based on the acquired profile data, the server uses generative AI to analyze the user's artistic preferences.

[0778] What it does: It feeds profile data into machine learning algorithms to identify user preferences.

[0779] Step 5:

[0780] Server: Generative AI generates an initial artwork based on the analysis results.

[0781] What it does: Generative AI generates digital art by combining appropriate art styles and elements, then converts the data into a VR-compatible format.

[0782] Step 6:

[0783] Server: Sends the generated artwork data to the device.

[0784] Specific operation: Digital art data is compressed and sent to the terminal using a high-speed communication protocol.

[0785] Step 7:

[0786] Terminal: Generates a VR space based on the received art data.

[0787] Specific operation: Using a 3D rendering engine, artwork is displayed in a VR space.

[0788] Step 8:

[0789] User: Experience the artwork in a VR space.

[0790] Specific Actions: Users interact with the artwork using eye tracking, gestures, voice input, and emotion recognition.

[0791] Step 9:

[0792] Device: Sends real-time reaction data from the user's gaze tracking, gesture recognition, voice input, and emotion recognition to the server.

[0793] Specific operation: All data acquired by the gaze tracking sensor, gesture recognition system, microphone, and emotion engine (facial expression analysis, tone of voice analysis) is analyzed and sent to the server in real time.

[0794] Step 10:

[0795] Server: Analyzes user reaction and emotional data, and the generative AI uses this data to evolve the artwork.

[0796] Specific behavior: Based on real-time data, the artwork is modified to add new elements or highlight parts that users like or that elicit an emotional response.

[0797] Step 11:

[0798] Server: The data of the evolved artwork is sent back to the device.

[0799] Specific behavior: Sends updated art data at high speed.

[0800] Step 12:

[0801] Device: Evolved artworks are reflected in the VR space in real time.

[0802] What it does: Instantly update artwork in the VR space and display new content to users.

[0803] Step 13:

[0804] User: After completing the exhibition experience, enter your evaluation in the feedback form.

[0805] Specific action: Fill out the feedback form that appears and provide your rating and comments about the art experience, then submit it.

[0806] Step 14:

[0807] Server: Stores user feedback data and provides it to the generation AI for next display optimization.

[0808] Specific operation: Feedback data is accumulated in a database and used as learning data for the generative AI.

[0809] Example 2

[0810] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0811] Conventional art exhibition systems have difficulty reflecting user preferences, reactions, and emotions in real time, making it difficult to provide a personalized art experience. Furthermore, there is a lack of mechanisms for incorporating user feedback into future exhibitions, making it difficult to provide a continuously optimized experience. To solve these issues, a system is needed that can acquire user profile data, reaction data, and emotional data in real time and evolve artworks based on that data.

[0812] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0813] In this invention, the server includes means for acquiring user attribute information, means for using a generation AI to analyze the user's artistic preferences based on the acquired attribute information, means for rendering artwork generated by the generation AI in a virtual reality space, means for acquiring data based on the user's gaze tracking and motion recognition in real time, means for analyzing the acquired data and using the generation AI to evolve the artwork, means for updating and reflecting the evolved artwork in the virtual reality space, and means for acquiring the user's emotional state in real time and analyzing the data, thereby enabling dynamic and personalized art exhibits based on the user's preferences, reactions, and emotions.

[0814] "User demographic information" is information that indicates a user's personal characteristics and preferences, such as the user's name, age, hobbies, and past activity.

[0815] "Generative AI" is an artificial intelligence technology used to generate or evolve artworks based on user attribute information and reaction data.

[0816] A "virtual reality space" is a three-dimensional digital environment that users access through a VR headset.

[0817] "Eye tracking" is a technology that detects a user's eye movements and gaze position in real time.

[0818] "Motion recognition" is a technology that detects a user's hand movements and body gestures and captures them as data.

[0819] "Real-time acquisition" means that data is collected almost simultaneously and processed immediately.

[0820] "Analysis" is the process of examining acquired data in detail and extracting meaning and patterns.

[0821] "Evolution" is the process of dynamically changing or improving a work of art based on user reactions and emotions.

[0822] "Emotional state" refers to the emotions such as joy, surprise, or anger that a user is feeling at a particular moment.

[0823] An "evaluation form" is an interface that allows users to enter their opinions and thoughts about artworks.

[0824] These definitions clearly describe each element of the system.

[0825] The present invention relates to a system for providing an interactive virtual reality space in which artworks are generated and evolved by recognizing the preferences, reactions, and emotions of users. Specific embodiments will be described below.

[0826] User Awareness and Initial Setup

[0827] server:

[0828] When a user logs in to the system, the server obtains the user's attribute information. This attribute information includes the user's name, age, artistic preferences, past reaction data, etc. For hardware, a general server or cloud infrastructure is used. For specific software, a database management system (e.g., MySQL, PostgreSQL) is used.

[0829] Device:

[0830] It detects when a user puts on a virtual reality (VR) headset and configures the VR environment, including calibrating the VR device (e.g., Oculus Quest 2) and setting up the user's position. The device configures the environment using dedicated VR software (e.g., Unity, Unreal Engine).

[0831] user:

[0832] Users put on a VR headset and controllers to begin accessing the system, and new users are asked to enter basic demographic information such as their name, age, and preferences.

[0833] Analysis of user preferences

[0834] server:

[0835] The acquired attribute information is analyzed and a generative AI is used to identify the user's artistic preferences. In this process, a generative AI model (e.g., GPT-3) is used to identify the user's preferred artistic style (abstract, figurative, color patterns, etc.). Specifically, the following prompt sentences are input to the AI ​​model:

[0836] "Generate abstract paintings that users love and incorporate past preferences."

[0837] Generating the initial artwork

[0838] server:

[0839] Based on the analysis results, the generative AI generates an initial artwork, which is then converted into a VR-compatible format (e.g., GLTF format) and sent to the device.

[0840] Building an interactive VR space

[0841] Device:

[0842] The VR space is constructed based on the artwork data sent from the server. In this process, VR platforms such as Unity and Unreal Engine are used to provide a three-dimensional interface that responds to the user's visual and auditory senses.

[0843] Real-time data capture and transmission

[0844] Device:

[0845] The system captures user actions (eye tracking, motion recognition, and voice input) in real time and sends the data to a server. For eye tracking, it uses a built-in camera and sensors, and for motion recognition, it uses a remote controller and hand tracking technology. For voice input, it uses a microphone, which is then analyzed by voice recognition software (e.g., Google Cloud Speech-to-Text).

[0846] Device: Using an emotion engine (e.g., Emotion API), emotional data is acquired from the user's facial expressions and voice and sent to the server. Specifically, facial expressions are analyzed from the user's camera footage, and the tone of voice is analyzed from microphone input.

[0847] Real-time reactions and artistic evolution

[0848] server:

[0849] The AI ​​analyzes user reaction and emotional data and uses this data to evolve the artwork. For example, if a user expresses interest in a particular color or shape, it will emphasize that part.

[0850] server:

[0851] By sending data of the evolved artwork to the device, it is instantly reflected in the VR space.

[0852] Device:

[0853] As new artworks are generated, existing works are updated in real time, providing a fresh user experience.

[0854] Feedback collection and optimization

[0855] server:

[0856] User feedback and ratings are collected and stored in the generative AI's learning database, which will further optimize the next art exhibit.

[0857] Device:

[0858] After the exhibition ends, a feedback form will be displayed to users, providing an interface where they can enter their ratings and comments.

[0859] Further processing for emotion recognition

[0860] Device:

[0861] The emotion engine captures emotional data from the user's facial expressions and voice in real time, including analyzing facial expressions from the user's camera footage and tone of voice from microphone input.

[0862] server:

[0863] The acquired emotional data is analyzed and the artwork is further adjusted based on the user's emotional state, for example by further emphasizing parts that made the user feel surprised or happy.

[0864] This system will enable personalized art exhibitions based on the user's preferences, reactions, and emotions, and will also provide a new art experience that transcends physical constraints through the use of digital technology.

[0865] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0866] Step 1:

[0867] When a user logs into the system, the server obtains the user's attribute information (name, age, artistic preferences, and past response data).

[0868] Input: User ID and login information

[0869] Data processing: Search the database using the user ID as a key and obtain the corresponding user information

[0870] Output: Retrieved user attribute information

[0871] Step 2:

[0872] The server analyzes the acquired attribute information and uses generative AI to identify the user's artistic preferences.

[0873] Input: User attribute information

[0874] Data calculation: Input prompts into the generative AI model and perform analysis that reflects user preferences.

[0875] Output: Information about the user's artistic preferences

[0876] Example prompt: "Generate abstract paintings that users will love and incorporate their past preferences."

[0877] Step 3:

[0878] The server uses generative AI to generate an initial artwork based on the user's preferences.

[0879] Input: Information about the user's artistic preferences

[0880] Data Computation: Generative AI model generates initial artwork based on prompt text

[0881] Output: Initial artwork data (converted to VR format)

[0882] Operation: The generated artwork data is converted into VR-compatible GLTF format and sent to the device.

[0883] Step 4:

[0884] The device creates a VR space based on the artwork data sent from the server.

[0885] Input: Initial artwork data in GLTF format

[0886] Data processing: Using VR software such as Unity or Unreal Engine, we create a three-dimensional VR space.

[0887] Output: The VR space experienced by the user

[0888] Action: Calibrate the VR device and set up the device to understand the user's position.

[0889] Step 5:

[0890] The device captures the user's gaze tracking, movement recognition, and voice input in real time within the VR space and sends the data to a server.

[0891] Input: User gaze, movement, and voice data

[0892] Data processing: Capture and process data using eye-tracking sensors, gesture recognition sensors, and voice recognition software

[0893] Output: Captured real-time data

[0894] How it works: Uses the emotion engine to get user emotion data and send it to the server

[0895] Step 6:

[0896] The server analyzes the real-time data it receives, and the generative AI uses this data to evolve the artwork.

[0897] Input: Captured real-time data and sentiment data

[0898] Data Computation: Generative AI models analyze real-time data and dynamically update artwork based on user responses

[0899] Output: Improved artwork data

[0900] How it works: If a particular color or shape shows interest, it highlights that area in near real time.

[0901] Step 7:

[0902] The server transmits data of the evolved artwork to the terminal.

[0903] Input: Evolved artwork data

[0904] Data processing: Reconverting the evolved artwork data into VR format

[0905] Output: Reconverted artwork data

[0906] Operation: Transmitted to the device and reflected instantly in the VR space

[0907] Step 8:

[0908] The device updates the existing VR space in real time as new artworks are created.

[0909] Input: Evolved artwork data

[0910] Data processing: Using VR software such as Unity or Unreal Engine, we update existing VR spaces with new artwork.

[0911] Output: Updated VR space

[0912] What it does: A completely new user experience

[0913] Step 9:

[0914] The server collects user feedback and ratings and stores them in the generative AI's learning database.

[0915] Input: User feedback and rating data

[0916] Data processing: Data collected through the feedback form is saved in the AI ​​learning database.

[0917] Output: Accumulated feedback data

[0918] Action: The display will be further optimized from next time onwards.

[0919] Step 10:

[0920] After the exhibition ends, the terminal displays a feedback form to users, providing an interface where they can enter their ratings and comments.

[0921] Input: User ratings and comments

[0922] Data processing: Evaluation data is collected using a feedback form and sent to the server.

[0923] Output: Collected assessment data

[0924] Behavior: Updates the interface presented to the user to prompt for feedback

[0925] In this way, by clarifying the specific actions, inputs and outputs performed at each step, an interactive VR artwork is provided that evolves in real time in response to the user's preferences, reactions and emotions.

[0926] (Application example 2)

[0927] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0928] Conventional art exhibition systems have struggled to generate personalized artworks that reflect users' preferences, reactions, and emotions in real time, and lacked a system for optimizing the next exhibition using feedback and emotional data, making it difficult to increase user satisfaction.

[0929] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0930] In this invention, the server includes means for acquiring user profile data, means for using a generation AI to analyze the user's art preferences based on the acquired profile data, means for rendering artwork generated by the generation AI in a virtual reality space, means for acquiring the user's gaze tracking, gesture recognition, voice input, and emotional data in real time, means for analyzing the acquired data and using the generation AI to evolve the artwork, means for updating and reflecting the evolved artwork in the virtual reality space, means for enhancing or adjusting the artwork based on the user's emotional data, and means for collecting user feedback and accumulating it in the generation AI's learning database, thereby enabling the provision of personalized artwork that reflects the user's preferences, reactions, and emotions in real time.

[0931] "User Profile Data" means personal information such as your name, age, art preferences, and past interaction data.

[0932] "Generative AI" is artificial intelligence that uses machine learning algorithms to generate and evolve artworks based on user data.

[0933] A "virtual reality space" is a three-dimensional interface space created using virtual reality technology that users can experience visually and aurally.

[0934] "Eye tracking" is a technology that captures and analyzes a user's eye movements in real time.

[0935] "Gesture recognition" is a technology that captures a user's hand and body movements and analyzes those movements.

[0936] "Voice input" is a technology that acquires and analyzes the user's voice data.

[0937] "Emotional data" is data that indicates the emotional state of a user, obtained from the user's facial expression, tone of voice, etc.

[0938] An "evolved artwork" is a work that has undergone some changes and adjustments to the original artwork based on user reaction and emotional data.

[0939] "Feedback" refers to information such as ratings and comments that users enter about artworks.

[0940] The "learning database" is a database that accumulates past data and feedback that the generative AI will use the next time it generates an artwork.

[0941] "Real time" is a time concept that refers to a system's immediate response and processing without delay.

[0942] This invention provides a system for providing an interactive virtual reality space in which artwork is generated and evolved based on the user's preferences, reactions, and emotions. Specific embodiments of this system are described below.

[0943] When a user logs into the system, the server retrieves profile data. This profile data includes the user's name, age, art preferences, and past reaction data. The device detects whether the user is wearing a smartphone or head-mounted display (HMD) and configures the environment. This includes calibrating the VR equipment and configuring the user's position in the environment. The user is required to enter basic profile information when accessing the system for the first time.

[0944] The server analyzes the acquired profile data and uses a generative AI to analyze the user's art preferences. This process uses machine learning algorithms to identify trends in the user's preferred art styles (abstract, figurative, color patterns, etc.). The generative AI generates an initial artwork based on the analysis results, and the data for this generated artwork is converted into a virtual reality-compatible format.

[0945] The device creates a virtual reality space based on the artwork data sent from the server, providing a three-dimensional interface that responds to the user's visual and auditory senses. Furthermore, the device captures the user's actions in the virtual reality space (eye tracking, gesture recognition, voice input, emotional data acquisition, etc.) in real time and sends the data to the server.

[0946] The server analyzes the user's reaction and emotional data. Based on the analyzed data, the generation AI evolves the artwork. Specifically, it makes changes such as emphasizing parts that the user particularly finds interesting. The data for the evolved artwork is then sent back to the device and instantly reflected in the virtual reality space. In this way, when a new artwork is generated, the existing artwork is updated in real time, completely renewing the user's experience.

[0947] The server collects user feedback and ratings and stores them in the generative AI's learning database. This feedback data is used to optimize the next exhibition. After the exhibition ends, the terminal displays a feedback form to the user, providing an interface where they can enter ratings and comments. The emotion engine obtains emotional data from the user's facial expressions and voice in real time. This includes analyzing facial expressions from the user's camera footage and voice tone from microphone input, and analyzes the obtained emotional data to further adjust the artwork based on their emotional state.

[0948] As a concrete example, consider a scenario where a user generates an artwork using a prompt such as, "Create a colorful abstract painting. In the past, the user has preferred blue and warm color patterns." The artwork generated based on this prompt evolves based on the user's response, e.g., emphasizing the area where the user's gaze is directed or changing color to reflect emotions. As a result, the user can enjoy a more personalized art experience.

[0949] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0950] Step 1:

[0951] A user logs into the system and enters profile data.

[0952] Input: User's name, age, art preferences, and past response data.

[0953] Output: User profile data.

[0954] Specific operation: A user accesses the system using a smartphone or head-mounted display (HMD) and enters profile information when logging in for the first time.

[0955] Step 2:

[0956] The server retrieves the user's profile data and uses generative AI to analyze the user's art preferences.

[0957] Input: The user's profile data obtained in step 1.

[0958] Output: Analysis of user's art preferences.

[0959] What it does: The server analyzes your profile data and uses machine learning algorithms to identify trends in your preferred art styles (abstract, figurative, color patterns, etc.).

[0960] Step 3:

[0961] Generative AI generates an initial artwork based on the analysis and converts it into a VR-compatible format.

[0962] Input: The analysis results from step 2.

[0963] Output: Virtual reality compatible artwork data.

[0964] Specific operation: The generation AI generates a prompt (e.g., "Generate a colorful abstract painting. Users have previously preferred blue and warm color patterns") and generates an artwork based on it. The generated data is converted into a VR format.

[0965] Step 4:

[0966] The device creates a virtual reality space based on the data of the artwork sent from the server.

[0967] Input: Virtual reality-ready artwork data generated in step 3.

[0968] Output: A work of art displayed in a virtual reality space.

[0969] How it works: When a user wears a head-mounted display (HMD), the device displays the artwork, providing the user with a three-dimensional interface that supports both visual and auditory senses.

[0970] Step 5:

[0971] The device captures the user's gaze tracking, gesture recognition, voice input, and emotional data in real time and transmits it to the server.

[0972] Input: User gaze, gesture, voice, and emotion data.

[0973] Output: User response data captured in real time.

[0974] How it works: The device uses the camera and microphone to analyze the user's facial expressions and tone of voice, capture eye tracking and gesture recognition data, and transmit this data to a server in real time.

[0975] Step 6:

[0976] The server analyzes user reaction data and uses generative AI to evolve the artwork.

[0977] Input: User response data obtained in step 5.

[0978] Output: Evolved artwork data.

[0979] How it works: The server analyzes the areas of interest and changes in emotions of the user, and the generative AI modifies the artwork based on that data, for example by emphasizing certain colors or shapes.

[0980] Step 7:

[0981] The device updates and reflects the evolved artwork in the virtual reality space.

[0982] Input: The evolved artwork data generated in step 6.

[0983] Output: An updated artwork in a virtual reality space.

[0984] What it does: The device updates the artwork already displayed in real time, providing the user with a new art experience.

[0985] Step 8:

[0986] The server collects user feedback and ratings and stores them in the generative AI's learning database.

[0987] Input: User feedback and ratings.

[0988] Output: Feedback data stored in the training database.

[0989] Specific operation: After the exhibition ends, the terminal displays a feedback form to the user. The user's ratings and comments are collected, and the server provides this to the generation AI as learning data.

[0990] Step 9:

[0991] The terminal obtains the user's emotional data in real time and further adjusts the artwork based on the user's emotional state.

[0992] Input: User facial and vocal emotion data.

[0993] Output: A coordinated piece of art.

[0994] How it works: The device analyzes the user's emotions based on data obtained from camera footage and microphone input, and then enhances or modifies the artwork based on that.

[0995] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0996] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0997] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0998] [Third embodiment]

[0999] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1000] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1001] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1002] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1003] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1004] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1005] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1006] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1007] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1008] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1009] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1010] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1011] This invention relates to a system for providing an interactive virtual reality (VR) space in which artwork is generated and evolved according to the user's preferences and reactions. Specific aspects of the system are described below.

[1012] User Awareness and Initial Setup

[1013] Server: When a user logs into the system, the server retrieves the user's profile data, which includes the user's name, age, art preferences, past reaction data, etc.

[1014] Device: Detects when the user puts on a VR headset and prepares the VR environment, including calibrating the VR equipment and configuring it to understand the user's position in the environment.

[1015] User: A user puts on a VR headset and controllers and begins accessing the system. First-time users are prompted to enter basic profile information.

[1016] Analysis of user preferences

[1017] Server: Analyzes user profile data and uses generative AI to analyze the user's art preferences. This process uses machine learning algorithms to identify trends in the user's preferred art styles (abstract, figurative, color patterns, etc.).

[1018] Server: The generative AI generates the initial artwork based on the analysis results. The data of this generated artwork is converted into a VR-compatible format.

[1019] Interactive VR space generation

[1020] Terminal: A VR space is constructed based on the artwork data sent from the server, providing a 3D interface that responds to the user's visual and auditory senses.

[1021] Device: Captures the user's actions in the VR space (eye tracking, gesture recognition, voice input, etc.) in real time and sends the data to the server.

[1022] Real-time reactions and artistic evolution

[1023] Server: The generative AI analyzes user reaction data and uses it to evolve the artwork, highlighting areas that the user particularly found interesting.

[1024] Server: Data of the evolved artwork is sent to the device, and is instantly reflected in the VR space.

[1025] Device: As new artworks are generated, existing artworks are updated in real time, completely transforming the user experience.

[1026] Feedback accumulation and optimization

[1027] Server: Collects user feedback and ratings and stores them in the generative AI's learning database. This feedback data is used to optimize the next exhibit.

[1028] Terminal: After the exhibition ends, a feedback form will be displayed to the user, providing an interface where they can enter their ratings and comments.

[1029] Specific examples

[1030] Example 1: When a new user accesses the system and enters their profile information, the generative AI generates an abstract painting in the VR space based on the user's preferences. If the user focuses their gaze on a particular color or shape, the artwork evolves to emphasize that aspect.

[1031] Example 2: When a repeat user revisits the system, artworks are generated that are more tailored to the user based on their past data, resulting in a more refined and personalized art experience for the user.

[1032] As described above, this system combines generative AI and VR technology to create dynamic art exhibits that respond to the user's preferences and reactions. By utilizing digital space, it can provide new art experiences that transcend physical constraints, and can also make a significant contribution to the fields of education and culture.

[1033] The processing flow will be explained below.

[1034] Step 1:

[1035] User: Accesses and logs into the system.

[1036] Specific operation: The user enters their ID and password and presses the login button.

[1037] Step 2:

[1038] Server: Retrieves the user's profile data.

[1039] Specific operation: Search and extract the user's past art viewing history, preferences, rating data, etc. from the database.

[1040] Step 3:

[1041] Device: Detects when the user is wearing a VR headset and configures the environment.

[1042] Specific operation: Calibrate the VR equipment and obtain the user's body position information.

[1043] Step 4:

[1044] Server: Based on the acquired profile data, the server uses generative AI to analyze the user's artistic preferences.

[1045] What it does: It feeds profile data into machine learning algorithms to identify user preferences.

[1046] Step 5:

[1047] Server: Generative AI generates an initial artwork based on the analysis results.

[1048] What it does: The AI ​​model generates digital art by combining appropriate art styles and elements, then converts the data into a VR-compatible format.

[1049] Step 6:

[1050] Server: Sends the generated artwork data to the device.

[1051] Specific operation: Digital art data is compressed and sent to the terminal using a high-speed communication protocol.

[1052] Step 7:

[1053] Terminal: Generates a VR space based on the received art data.

[1054] Specific operation: Using a 3D rendering engine, artwork is displayed in a VR space.

[1055] Step 8:

[1056] User: Experience the artwork in a VR space.

[1057] Specific actions: Users interact with the artwork using eye tracking and gestures.

[1058] Step 9:

[1059] Device: Sends real-time reaction data from the user's gaze tracking and gesture recognition to the server.

[1060] Specific operation: Data acquired by the gaze tracking sensor and gesture recognition system is analyzed and sent to the server in real time.

[1061] Step 10:

[1062] Server: Analyzes user reaction data, and the generative AI uses this data to evolve the artwork.

[1063] Specific behavior: Based on real-time data, the system makes changes to the artwork, such as adding new elements or emphasizing parts that users like.

[1064] Step 11:

[1065] Server: The data of the evolved artwork is sent back to the device.

[1066] Specific behavior: Sends updated art data at high speed.

[1067] Step 12:

[1068] Device: Evolved artworks are reflected in the VR space in real time.

[1069] What it does: Instantly update artwork in the VR space and display new content to users.

[1070] Step 13:

[1071] User: After completing the exhibition experience, enter your evaluation in the feedback form.

[1072] Specific action: Fill out the feedback form that appears and provide your rating and comments about the art experience, then submit it.

[1073] Step 14:

[1074] Server: Stores user feedback data and provides it to the generation AI for next display optimization.

[1075] Specific operation: Feedback data is accumulated in a database and used as learning data for the generative AI.

[1076] Example 1

[1077] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1078] Conventional art exhibition systems have struggled to dynamically generate and evolve artworks based on individual user preferences and real-time responses. They also have limited means to effectively incorporate interactive data, such as user gaze tracking and gesture recognition. As a result, the experience for each user is not fully personalized, and there is no appropriate mechanism for incorporating user feedback into future exhibitions.

[1079] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1080] In this invention, the server includes means for acquiring user profile data, means for using artificial intelligence to analyze user preferences based on the acquired user profile data, means for rendering a digital artwork generated by the artificial intelligence in a virtual reality space, means for acquiring data based on the user's gaze tracking and motion recognition in real time, means for analyzing the acquired data and using artificial intelligence to evolve the digital artwork, and means for updating and reflecting the evolved digital artwork in the virtual reality space, thereby enabling a dynamic and personalized art experience that reflects individual user preferences and real-time reactions.

[1081] "User profile data" refers to any information related to a user who logs into the system, such as the user's name, age, art preferences, and past response data.

[1082] "Artificial intelligence" refers to technologies including machine learning algorithms and deep learning models that analyze user profile data and real-time reaction data to generate and evolve appropriate artworks.

[1083] "Digital artwork" refers to art generated by artificial intelligence, the data of which is converted into a format (e.g., FBX or OBJ) that can be rendered in a virtual reality space.

[1084] "Virtual reality space" refers to a three-dimensional digital environment that users can immerse themselves in using devices such as VR headsets.

[1085] "Eye tracking" refers to a technology that tracks the direction a user is looking or a specific object in a virtual reality space in real time.

[1086] "Motion recognition" refers to the technology of capturing a user's gestures and body movements with sensors and analyzing them.

[1087] "Real-time data" refers to data obtained through user gaze tracking and / or motion recognition that is processed and analyzed in real time.

[1088] MODE FOR CARRYING OUT THE INVENTION

[1089] This invention relates to a system that provides an interactive virtual reality (VR) space in which digital artwork is generated and evolved according to the preferences and reactions of users. Specific embodiments will be described below.

[1090] User Awareness and Initial Setup

[1091] Server: When a user logs into the system, the server retrieves the user's profile data (such as name, age, art preferences, past reaction data, etc.) This profile data is retrieved using database queries, and the data is cleansed and analyzed using Python libraries (e.g., Pandas, NumPy).

[1092] Device: When a user puts on a VR headset (e.g., a general-purpose VR headset), the device detects this information and prepares the VR environment, including calibrating the VR device and configuring it to understand the user's position (e.g., 6DOF tracking).

[1093] User: The user puts on the VR headset and controllers and begins accessing the system. First-time users are asked to enter basic profile information (such as name, age, and art preferences) through a form in the interface.

[1094] Analysis of user preferences

[1095] Server: The server analyzes the acquired user profile data and uses generative AI to analyze the user's art preferences. In this process, machine learning algorithms (e.g., TensorFlow, PyTorch) are used to identify trends in the user's preferred art styles (abstract art, figurative art, color patterns, etc.).

[1096] Server: Generative AI (e.g., GPT-3, GANs) generates the initial digital artwork based on the analysis results. The Assimp library is used to convert this generated artwork data into a VR-compatible format (e.g., FBX format or OBJ format).

[1097] Interactive VR space generation

[1098] Terminal: Using Unity or Unreal Engine, a VR space is created based on the artwork data sent from the server. This provides a 3D interface that responds to the user's vision and hearing.

[1099] Device: Configure it using the necessary APIs (e.g., OpenVR SDK, general-purpose VR SDK) to capture the user's actions in the VR space (e.g., eye tracking, motion recognition, voice input, etc.) in real time.

[1100] Real-time reactions and artistic evolution

[1101] Server: Analyzes the user's real-time reaction data (eye tracking data, motion recognition data, etc.), and the generative AI evolves the artwork based on this. Specifically, computer vision technology (e.g., OpenCV) is used to analyze the gaze data.

[1102] Server: Sends the evolved artwork data to the device. This process uses WebSocket to exchange data in real time.

[1103] Device: When a new artwork is generated, the existing artwork is instantly updated in the VR space, leveraging the real-time rendering capabilities of Unity and Unreal Engine, so the changes are instantly reflected in the user's vision.

[1104] Feedback accumulation and optimization

[1105] Server: After the exhibition, the feedback and evaluations collected from users are stored in the AI's learning database. This data will be used for future art generation.

[1106] Terminal: After the exhibition ends, a feedback form is displayed to the user, providing an interface where they can enter their ratings and comments. This form data is collected using HTML and JavaScript and sent to the server.

[1107] Specific examples

[1108] Example 1:

[1109] Server: When a new user logs in, it collects profile information and queries the database. Python is used to clean the data and extract preferences. The generative AI generates abstract art based on the data and converts it into FBX format.

[1110] Device: Receives artwork data and displays it in the VR space using Unity. When the user directs their gaze at a specific color or shape, eye-tracking data is sent to the server.

[1111] Server: To evolve the artwork based on the eye-tracking data, the generative AI updates the artwork design. It converts the new data into a VR format and sends it to the device.

[1112] Device: The new artwork is instantly updated in the VR space, allowing users to experience the changes in real time.

[1113] Example 2:

[1114] Server: When a repeat user accesses the site again, the initial settings are made based on the user's past profile data and responses, and an art piece is generated that is further tailored to the user.

[1115] Device: Compares existing art with new art and dynamically updates the VR space. When the user responds with gestures to specific shapes, the information is sent to the server in real time.

[1116] Server: The artwork is partially evolved based on gesture recognition data and the information is sent to the device.

[1117] On your device: Display updated artwork so that your art experience is more refined and personalized than the last time.

[1118] Prompt Sentence Examples

[1119] "Describe the process of generating abstract paintings based on user profile information and analyzing the user's eye-tracking data to evolve the artwork."

[1120] As described above, this system combines generative AI and VR technology to provide a dynamic art experience that responds to the user's preferences and reactions. By utilizing digital space, it can provide new art experiences that transcend physical constraints and make significant contributions to the fields of education and culture.

[1121] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1122] Program processing flow

[1123] Step 1: User Awareness and Initial Setup

[1124] Server: When a user logs in to the system, the server retrieves the user's profile data (such as name, age, art preferences, and past reaction data) using a database query and cleanses the data using Python libraries (e.g., Pandas and NumPy).

[1125] Input: User login information.

[1126] Output: Cleansed user profile data.

[1127] Device: When the user puts on the VR headset, the device detects this information and prepares the VR environment. Specifically, it calibrates the VR equipment and sets up the user's position (for example, 6DOF tracking).

[1128] Input: VR headset wearing information.

[1129] Output: Calibrated VR environment setup.

[1130] User: The user puts on the VR headset and controllers and begins accessing the system. First-time users enter basic profile information (such as name, age, and art preferences) through a form in the interface.

[1131] Input: Basic profile information.

[1132] Output: The entered profile information.

[1133] Step 2: Analyzing user preferences

[1134] Server: The server analyzes user profile data and uses generative AI (e.g., TensorFlow or PyTorch) to analyze the user's art preferences. It uses machine learning algorithms to identify trends in the user's preferred art styles (e.g., abstract, figurative, color patterns, etc.).

[1135] Input: Cleansed user profile data.

[1136] Output: User's art preference information.

[1137] Server: The generative AI generates the initial digital artwork based on the analysis results. The generated artwork data is converted into a VR-compatible format (e.g., FBX or OBJ format) using the Assimp library.

[1138] Input: User's art preference information.

[1139] Output: Digital artwork data in a VR-compatible format.

[1140] Step 3: Creating an interactive VR space

[1141] Device: Using Unity or Unreal Engine, a VR space is created based on the artwork data sent from the server, providing a 3D interface that responds to the user's visual and auditory senses.

[1142] Input: Digital artwork data in a VR-compatible format.

[1143] Output: The constructed VR space.

[1144] Device: Captures the user's actions in the VR space (eye tracking, motion recognition, voice input, etc.) in real time and sends the data to the server. Uses the required API (e.g., OpenVR SDK, general-purpose VR SDK).

[1145] Input: User operation data.

[1146] Output: Operation data sent to the server.

[1147] Step 4: Real-time reactions and artistic evolution

[1148] Server: Analyzes the user's real-time reaction data (eye tracking data, motion recognition data, etc.), and the generative AI evolves the artwork based on this data. Computer vision technology (e.g., OpenCV) is used to analyze the gaze data.

[1149] Input: Real-time user response data.

[1150] Output: Evolved digital artwork data.

[1151] Server: Sends the evolved artwork data to the device. Data is exchanged in real time using WebSocket.

[1152] Input: Evolved digital artwork data.

[1153] Output: Artwork data sent to the device.

[1154] Device: As new artworks are generated, existing artworks are updated in real time within the VR space, utilizing the real-time rendering capabilities of Unity and Unreal Engine.

[1155] Input: Evolved digital artwork data.

[1156] Output: Updated VR space.

[1157] Step 5: Gather feedback and optimize

[1158] Server: After the exhibition is over, the feedback and evaluations collected from users are stored in the Generative AI's learning database. This data is used to optimize the next exhibition.

[1159] Input: User feedback data.

[1160] Output: Saved feedback data.

[1161] Terminal: After the exhibition ends, a feedback form is displayed to the user, providing an interface where they can enter their ratings and comments. The form data is collected using HTML and JavaScript and sent to the server.

[1162] Input: User ratings and comments.

[1163] Output: The feedback form data sent to the server.

[1164] Through these processing steps, the system provides a dynamic art experience that responds to user preferences and real-time responses.

[1165] (Application example 1)

[1166] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1167] Conventional entertainment systems for autonomous vehicles have the drawback of being difficult to provide content that responds to individual user preferences and real-time responses, and are difficult to provide an experience with enhanced visual and tactile interactivity due to the limited environment inside the vehicle.

[1168] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1169] In this invention, the server includes means for acquiring user profile data, means for using a generation AI to analyze the user's art preferences based on the acquired profile data, means for rendering artwork generated by the generation AI in a virtual reality space, means for acquiring data based on the user's gaze tracking and gesture recognition in real time, means for analyzing the acquired data and using the generation AI to evolve the artwork, means for updating and reflecting the evolved artwork in the virtual reality space, and means for providing a terminal for using these means within an autonomous vehicle, thereby enabling the provision of a high-quality, personalized art experience in real time even within an autonomous vehicle.

[1170] "User Profile Data" means data that includes personal information about you, such as your name, age, and artistic preferences.

[1171] "Generative AI" is artificial intelligence that uses machine learning algorithms to generate and evolve artworks based on user preferences.

[1172] A "virtual reality space" is a three-dimensional virtual space that provides users with an immersive experience through visual and auditory interfaces.

[1173] "Eye tracking" is a technology that detects and analyzes the user's eye movements in real time.

[1174] "Gesture recognition" is a technology that uses cameras and sensors to detect and analyze the movements of a user's hands and fingers.

[1175] "Real-time acquisition means" refers to a system or device that instantly collects interaction data such as user gaze and gestures.

[1176] An "autonomous vehicle" is a vehicle that drives autonomously using artificial intelligence and sensor technology.

[1177] A "terminal" is a device or system (e.g., a head-mounted display) that allows a user to access a virtual reality space.

[1178] A "feedback form" is an interface that allows users to enter their opinions and evaluations of artworks.

[1179] The "generative AI learning database" is a database used to accumulate user feedback and reaction data and improve the performance of the generative AI.

[1180] The embodiments of the present invention will be specifically described below.

[1181] Retrieving a user profile

[1182] When a user logs into the system, the server retrieves the user's profile data, which includes the user's name, age, art preferences, past interaction data, etc. This provides the basis for collecting data to provide a personalized experience for each user.

[1183] User Awareness and Initial Setup

[1184] The device detects when the user is wearing a VR headset and prepares the VR environment, including calibrating the head-mounted display (e.g., Oculus Quest) and configuring it to understand the user's position. When a user first accesses the system, they are required to enter basic profile information.

[1185] Art taste analysis

[1186] The server analyzes the profile data and uses generative AI to analyze the user's art preferences. This process uses machine learning algorithms to identify trends in the user's preferred art styles (abstract, figurative, color patterns, etc.).

[1187] Generating and Rendering Initial Artwork

[1188] The generative AI generates an initial artwork based on the user's preferences. The data for this artwork is converted into a format compatible with the virtual reality space and sent to the device. The device then creates a virtual reality space based on the data sent from the server. This provides a 3D interface that responds to the user's visual and auditory senses.

[1189] Interactive operation and real-time updates

[1190] The device captures the user's actions in the virtual reality space (eye tracking, gesture recognition, voice input, etc.) in real time and sends the data to the server. The server analyzes the user's reaction data, and the generative AI uses this data to evolve the artwork. Changes are made, such as emphasizing areas that the user has shown particular interest in. Data on the evolved artwork is sent to the device and instantly reflected in the virtual reality space, completely transforming the user's experience.

[1191] Feedback collection and optimization

[1192] The server collects user feedback and ratings and stores them in the generative AI's learning database. This feedback data is used to optimize the next exhibit. After the exhibit is over, the terminal displays a feedback form to the user, providing an interface where they can enter ratings and comments.

[1193] Use in self-driving vehicles

[1194] This system is installed in autonomous vehicles, allowing users to enjoy art experiences using VR headsets while traveling. The in-car device captures the user's actions in real time and works with a server to evolve the artwork, providing immersive entertainment even while traveling.

[1195] Examples:

[1196] Example 1: A new user accesses the system and enters their profile information. Generative AI generates an abstract painting in a virtual reality space based on the user's preferences. If the user focuses their gaze on a particular color or shape, the artwork evolves to emphasize that feature.

[1197] Example 2: When a returning user revisits the system, the system generates more personalized artwork based on their past data, resulting in a more refined and personalized art experience.

[1198] Example prompt sentence:

[1199] "Develop an app for autonomous vehicles that allows passengers to use VR headsets to experience and evolve art according to their preferences."

[1200] In this way, this invention combines generative AI and virtual reality technology to create dynamic art exhibits that respond to user preferences and reactions, providing a new entertainment experience even inside autonomous vehicles.

[1201] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1202] Step 1:

[1203] When a user logs in to the system, the server retrieves the user's profile data. The profile data includes the user's name, age, art preferences, and past response data. This provides the basis for the user to receive a personalized experience. The input is the user's login information, and the output is the user's profile data.

[1204] Step 2:

[1205] The device detects when the user puts on a VR headset and prepares the virtual reality environment. This includes calibrating the head-mounted display (e.g., Oculus Quest) and determining the user's position. The input is the state of the VR headset and user position information, and the output is information that the virtual reality environment is ready.

[1206] Step 3:

[1207] The server analyzes the acquired profile data and uses generative AI to analyze the user's art preferences. Machine learning algorithms are used to identify trends in the user's preferred art styles (abstract, figurative, color patterns, etc.). The input is the profile data, and the output is the analysis of the user's art preferences.

[1208] Step 4:

[1209] The server uses a generative AI to generate an initial artwork based on the user's preferences, converts it into a format compatible with the virtual reality space, and sends it to the device. The input is the user's art preference analysis results, and the output is VR-compatible initial artwork data.

[1210] Step 5:

[1211] The device constructs a virtual reality space based on the artwork data received from the server, providing a 3D interface that responds to the user's visual and auditory senses. The input is the initial artwork data, and the output is the constructed virtual reality space.

[1212] Step 6:

[1213] The device captures the user's actions in the virtual reality space (eye tracking, gesture recognition, voice input) in real time and sends the data to the server. The input is the user's action data, and the output is the data sent to the server.

[1214] Step 7:

[1215] The server analyzes the user's operation data, and the generative AI evolves the artwork based on this data. In particular, it makes changes such as emphasizing the parts that the user showed interest in. The input is the user's operation data, and the output is the evolved artwork data.

[1216] Step 8:

[1217] The server sends the evolved artwork data to the device, which then instantly reflects it in the virtual reality space. The input is the evolved artwork data, and the output is the updated virtual reality space.

[1218] Step 9:

[1219] The server collects user feedback and ratings and stores them in the generative AI's learning database. This feedback data is used to optimize the next display. The input is user feedback, and the output is an updated learning database.

[1220] Step 10:

[1221] The terminal displays a feedback form to the user after the exhibition has ended, providing an interface where users can enter their ratings and comments. The input triggers the end of the exhibition, and the output is the feedback information from the user.

[1222] The entire process enables a personalized art experience to be delivered in real time within the autonomous vehicle, and the system dynamically adapts to the user's preferences and reactions, delivering high-quality, immersive entertainment.

[1223] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1224] This invention relates to a system for providing an interactive virtual reality (VR) space that generates and evolves artwork by recognizing the user's preferences, reactions, and emotions. Specific aspects of the system are described below.

[1225] User Awareness and Initial Setup

[1226] Server: When a user logs into the system, the server retrieves the user's profile data, which includes the user's name, age, art preferences, past reaction data, etc.

[1227] Device: Detects when the user puts on a VR headset and configures the environment, including calibrating the VR equipment and locating the user in the environment.

[1228] User: A user puts on a VR headset and controllers and begins accessing the system. First-time users are prompted to enter basic profile information.

[1229] Analysis of user preferences

[1230] Server: Analyzes user profile data and uses generative AI to analyze the user's art preferences. This process uses machine learning algorithms to identify trends in the user's preferred art styles (abstract, figurative, color patterns, etc.).

[1231] Server: The generative AI generates the initial artwork based on the analysis results. The data of this generated artwork is converted into a VR-compatible format.

[1232] Interactive VR space generation

[1233] Terminal: A VR space is constructed based on the artwork data sent from the server, providing a 3D interface that responds to the user's visual and auditory senses.

[1234] Terminal: Captures the user's actions in the VR space (eye tracking, gesture recognition, voice input, emotion recognition using the emotion engine, etc.) in real time and sends the data to the server.

[1235] Real-time reactions and artistic evolution

[1236] Server: The generative AI analyzes the user's reaction and emotional data and uses this data to evolve the artwork, highlighting areas that the user particularly found interesting.

[1237] Server: Data of the evolved artwork is sent to the device, and is instantly reflected in the VR space.

[1238] Device: As new artworks are generated, existing artworks are updated in real time, completely transforming the user experience.

[1239] Feedback accumulation and optimization

[1240] Server: Collects user feedback and ratings and stores them in the generative AI's learning database. This feedback data is used to optimize the next exhibit.

[1241] Terminal: After the exhibition ends, a feedback form will be displayed to the user, providing an interface where they can enter their ratings and comments.

[1242] Further processing for emotion recognition

[1243] On the device: The emotion engine captures emotional data from the user's facial expressions and voice in real time, including analyzing facial expressions from the user's camera footage and tone of voice from the microphone input.

[1244] Server: Analyzes the acquired emotional data and further adjusts the artwork based on the user's emotional state, such as further emphasizing parts that made the user feel surprised or happy.

[1245] Specific examples

[1246] Example 1: When a new user accesses the system and enters their profile information, the generative AI generates an abstract painting in the VR space based on the user's preferences. If the user focuses their gaze on a particular color or shape, and the emotion engine detects the user's joy, the artwork evolves to emphasize that part.

[1247] Example 2: When a repeat user revisits the system, a more personalized artwork is generated based on their past and emotional data, resulting in a more refined and personalized art experience for the user.

[1248] Summary of implementation procedures

[1249] The system combines generative AI, VR technology, eye tracking, gesture recognition, voice input, and emotion recognition to create dynamic and personalized art exhibits that respond to the user's preferences, reactions, and emotions. By utilizing digital space, it can provide new art experiences that transcend physical constraints, and make significant contributions to the fields of education and culture.

[1250] The processing flow will be explained below.

[1251] Step 1:

[1252] User: Accesses and logs into the system.

[1253] Specific operation: The user enters their ID and password and presses the login button.

[1254] Step 2:

[1255] Server: Retrieves the user's profile data.

[1256] Specific operation: Search and extract the user's past art viewing history, preferences, rating data, etc. from the database.

[1257] Step 3:

[1258] Device: Detects when the user is wearing a VR headset and configures the environment.

[1259] Specific operation: Calibrate the VR equipment and obtain the user's body position information.

[1260] Step 4:

[1261] Server: Based on the acquired profile data, the server uses generative AI to analyze the user's artistic preferences.

[1262] What it does: It feeds profile data into machine learning algorithms to identify user preferences.

[1263] Step 5:

[1264] Server: Generative AI generates an initial artwork based on the analysis results.

[1265] What it does: Generative AI generates digital art by combining appropriate art styles and elements, then converts the data into a VR-compatible format.

[1266] Step 6:

[1267] Server: Sends the generated artwork data to the device.

[1268] Specific operation: Digital art data is compressed and sent to the terminal using a high-speed communication protocol.

[1269] Step 7:

[1270] Terminal: Generates a VR space based on the received art data.

[1271] Specific operation: Using a 3D rendering engine, artwork is displayed in a VR space.

[1272] Step 8:

[1273] User: Experience the artwork in a VR space.

[1274] Specific Actions: Users interact with the artwork using eye tracking, gestures, voice input, and emotion recognition.

[1275] Step 9:

[1276] Device: Sends real-time reaction data from the user's gaze tracking, gesture recognition, voice input, and emotion recognition to the server.

[1277] Specific operation: All data acquired by the gaze tracking sensor, gesture recognition system, microphone, and emotion engine (facial expression analysis, tone of voice analysis) is analyzed and sent to the server in real time.

[1278] Step 10:

[1279] Server: Analyzes user reaction and emotional data, and the generative AI uses this data to evolve the artwork.

[1280] Specific behavior: Based on real-time data, the artwork is modified to add new elements or highlight parts that users like or that elicit an emotional response.

[1281] Step 11:

[1282] Server: The data of the evolved artwork is sent back to the device.

[1283] Specific behavior: Sends updated art data at high speed.

[1284] Step 12:

[1285] Device: Evolved artworks are reflected in the VR space in real time.

[1286] What it does: Instantly update artwork in the VR space and display new content to users.

[1287] Step 13:

[1288] User: After completing the exhibition experience, enter your evaluation in the feedback form.

[1289] Specific action: Fill out the feedback form that appears and provide your rating and comments about the art experience, then submit it.

[1290] Step 14:

[1291] Server: Stores user feedback data and provides it to the generation AI for next display optimization.

[1292] Specific operation: Feedback data is accumulated in a database and used as learning data for the generative AI.

[1293] Example 2

[1294] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1295] Conventional art exhibition systems have difficulty reflecting user preferences, reactions, and emotions in real time, making it difficult to provide a personalized art experience. Furthermore, there is a lack of mechanisms for incorporating user feedback into future exhibitions, making it difficult to provide a continuously optimized experience. To solve these issues, a system is needed that can acquire user profile data, reaction data, and emotional data in real time and evolve artworks based on that data.

[1296] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1297] In this invention, the server includes means for acquiring user attribute information, means for using a generation AI to analyze the user's artistic preferences based on the acquired attribute information, means for rendering artwork generated by the generation AI in a virtual reality space, means for acquiring data based on the user's gaze tracking and motion recognition in real time, means for analyzing the acquired data and using the generation AI to evolve the artwork, means for updating and reflecting the evolved artwork in the virtual reality space, and means for acquiring the user's emotional state in real time and analyzing the data, thereby enabling dynamic and personalized art exhibits based on the user's preferences, reactions, and emotions.

[1298] "User demographic information" is information that indicates a user's personal characteristics and preferences, such as the user's name, age, hobbies, and past activity.

[1299] "Generative AI" is an artificial intelligence technology used to generate or evolve artworks based on user attribute information and reaction data.

[1300] A "virtual reality space" is a three-dimensional digital environment that users access through a VR headset.

[1301] "Eye tracking" is a technology that detects a user's eye movements and gaze position in real time.

[1302] "Motion recognition" is a technology that detects a user's hand movements and body gestures and captures them as data.

[1303] "Real-time acquisition" means that data is collected almost simultaneously and processed immediately.

[1304] "Analysis" is the process of examining acquired data in detail and extracting meaning and patterns.

[1305] "Evolution" is the process of dynamically changing or improving a work of art based on user reactions and emotions.

[1306] "Emotional state" refers to the emotions such as joy, surprise, or anger that a user is feeling at a particular moment.

[1307] An "evaluation form" is an interface that allows users to enter their opinions and thoughts about artworks.

[1308] These definitions clearly describe each element of the system.

[1309] The present invention relates to a system for providing an interactive virtual reality space in which artworks are generated and evolved by recognizing the preferences, reactions, and emotions of users. Specific embodiments will be described below.

[1310] User Awareness and Initial Setup

[1311] server:

[1312] When a user logs in to the system, the server obtains the user's attribute information. This attribute information includes the user's name, age, artistic preferences, past reaction data, etc. For hardware, a general server or cloud infrastructure is used. For specific software, a database management system (e.g., MySQL, PostgreSQL) is used.

[1313] Device:

[1314] It detects when a user puts on a virtual reality (VR) headset and configures the VR environment, including calibrating the VR device (e.g., Oculus Quest 2) and setting up the user's position. The device configures the environment using dedicated VR software (e.g., Unity, Unreal Engine).

[1315] user:

[1316] Users put on a VR headset and controllers to begin accessing the system, and new users are asked to enter basic demographic information such as their name, age, and preferences.

[1317] Analysis of user preferences

[1318] server:

[1319] The acquired attribute information is analyzed and a generative AI is used to identify the user's artistic preferences. In this process, a generative AI model (e.g., GPT-3) is used to identify the user's preferred artistic style (abstract, figurative, color patterns, etc.). Specifically, the following prompt sentences are input to the AI ​​model:

[1320] "Generate abstract paintings that users love and incorporate past preferences."

[1321] Generating the initial artwork

[1322] server:

[1323] Based on the analysis results, the generative AI generates an initial artwork, which is then converted into a VR-compatible format (e.g., GLTF format) and sent to the device.

[1324] Building an interactive VR space

[1325] Device:

[1326] The VR space is constructed based on the artwork data sent from the server. In this process, VR platforms such as Unity and Unreal Engine are used to provide a three-dimensional interface that responds to the user's visual and auditory senses.

[1327] Real-time data capture and transmission

[1328] Device:

[1329] The system captures user actions (eye tracking, motion recognition, and voice input) in real time and sends the data to a server. For eye tracking, it uses a built-in camera and sensors, and for motion recognition, it uses a remote controller and hand tracking technology. For voice input, it uses a microphone, which is then analyzed by voice recognition software (e.g., Google Cloud Speech-to-Text).

[1330] Device: Using an emotion engine (e.g., Emotion API), emotional data is acquired from the user's facial expressions and voice and sent to the server. Specifically, facial expressions are analyzed from the user's camera footage, and the tone of voice is analyzed from microphone input.

[1331] Real-time reactions and artistic evolution

[1332] server:

[1333] The AI ​​analyzes user reaction and emotional data and uses this data to evolve the artwork. For example, if a user expresses interest in a particular color or shape, it will emphasize that part.

[1334] server:

[1335] By sending data of the evolved artwork to the device, it is instantly reflected in the VR space.

[1336] Device:

[1337] As new artworks are generated, existing works are updated in real time, providing a fresh user experience.

[1338] Feedback collection and optimization

[1339] server:

[1340] User feedback and ratings are collected and stored in the generative AI's learning database, which will further optimize the next art exhibit.

[1341] Device:

[1342] After the exhibition ends, a feedback form will be displayed to users, providing an interface where they can enter their ratings and comments.

[1343] Further processing for emotion recognition

[1344] Device:

[1345] The emotion engine captures emotional data from the user's facial expressions and voice in real time, including analyzing facial expressions from the user's camera footage and tone of voice from microphone input.

[1346] server:

[1347] The acquired emotional data is analyzed and the artwork is further adjusted based on the user's emotional state, for example by further emphasizing parts that made the user feel surprised or happy.

[1348] This system will enable personalized art exhibitions based on the user's preferences, reactions, and emotions, and will also provide a new art experience that transcends physical constraints through the use of digital technology.

[1349] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1350] Step 1:

[1351] When a user logs into the system, the server obtains the user's attribute information (name, age, artistic preferences, and past response data).

[1352] Input: User ID and login information

[1353] Data processing: Search the database using the user ID as a key and obtain the corresponding user information

[1354] Output: Retrieved user attribute information

[1355] Step 2:

[1356] The server analyzes the acquired attribute information and uses generative AI to identify the user's artistic preferences.

[1357] Input: User attribute information

[1358] Data calculation: Input prompts into the generative AI model and perform analysis that reflects user preferences.

[1359] Output: Information about the user's artistic preferences

[1360] Example prompt: "Generate abstract paintings that users will love and incorporate their past preferences."

[1361] Step 3:

[1362] The server uses generative AI to generate an initial artwork based on the user's preferences.

[1363] Input: Information about the user's artistic preferences

[1364] Data Computation: Generative AI model generates initial artwork based on prompt text

[1365] Output: Initial artwork data (converted to VR format)

[1366] Operation: The generated artwork data is converted into VR-compatible GLTF format and sent to the device.

[1367] Step 4:

[1368] The device creates a VR space based on the artwork data sent from the server.

[1369] Input: Initial artwork data in GLTF format

[1370] Data processing: Using VR software such as Unity or Unreal Engine, we create a three-dimensional VR space.

[1371] Output: The VR space experienced by the user

[1372] Action: Calibrate the VR device and set up the device to understand the user's position.

[1373] Step 5:

[1374] The device captures the user's gaze tracking, movement recognition, and voice input in real time within the VR space and sends the data to a server.

[1375] Input: User gaze, movement, and voice data

[1376] Data processing: Capture and process data using eye-tracking sensors, gesture recognition sensors, and voice recognition software

[1377] Output: Captured real-time data

[1378] How it works: Uses the emotion engine to get user emotion data and send it to the server

[1379] Step 6:

[1380] The server analyzes the real-time data it receives, and the generative AI uses this data to evolve the artwork.

[1381] Input: Captured real-time data and sentiment data

[1382] Data Computation: Generative AI models analyze real-time data and dynamically update artwork based on user responses

[1383] Output: Improved artwork data

[1384] How it works: If a particular color or shape shows interest, it highlights that area in near real time.

[1385] Step 7:

[1386] The server transmits data of the evolved artwork to the terminal.

[1387] Input: Evolved artwork data

[1388] Data processing: Reconverting the evolved artwork data into VR format

[1389] Output: Reconverted artwork data

[1390] Operation: Transmitted to the device and reflected instantly in the VR space

[1391] Step 8:

[1392] The device updates the existing VR space in real time as new artworks are created.

[1393] Input: Evolved artwork data

[1394] Data processing: Using VR software such as Unity or Unreal Engine, we update existing VR spaces with new artwork.

[1395] Output: Updated VR space

[1396] What it does: A completely new user experience

[1397] Step 9:

[1398] The server collects user feedback and ratings and stores them in the generative AI's learning database.

[1399] Input: User feedback and rating data

[1400] Data processing: Data collected through the feedback form is saved in the AI ​​learning database.

[1401] Output: Accumulated feedback data

[1402] Action: The display will be further optimized from next time onwards.

[1403] Step 10:

[1404] After the exhibition ends, the terminal displays a feedback form to users, providing an interface where they can enter their ratings and comments.

[1405] Input: User ratings and comments

[1406] Data processing: Evaluation data is collected using a feedback form and sent to the server.

[1407] Output: Collected assessment data

[1408] Behavior: Updates the interface presented to the user to prompt for feedback

[1409] In this way, by clarifying the specific actions, inputs and outputs performed at each step, an interactive VR artwork is provided that evolves in real time in response to the user's preferences, reactions and emotions.

[1410] (Application example 2)

[1411] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1412] Conventional art exhibition systems have struggled to generate personalized artworks that reflect users' preferences, reactions, and emotions in real time, and lacked a system for optimizing the next exhibition using feedback and emotional data, making it difficult to increase user satisfaction.

[1413] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1414] In this invention, the server includes means for acquiring user profile data, means for using a generation AI to analyze the user's art preferences based on the acquired profile data, means for rendering artwork generated by the generation AI in a virtual reality space, means for acquiring the user's gaze tracking, gesture recognition, voice input, and emotional data in real time, means for analyzing the acquired data and using the generation AI to evolve the artwork, means for updating and reflecting the evolved artwork in the virtual reality space, means for enhancing or adjusting the artwork based on the user's emotional data, and means for collecting user feedback and accumulating it in the generation AI's learning database, thereby enabling the provision of personalized artwork that reflects the user's preferences, reactions, and emotions in real time.

[1415] "User Profile Data" means personal information such as your name, age, art preferences, and past interaction data.

[1416] "Generative AI" is artificial intelligence that uses machine learning algorithms to generate and evolve artworks based on user data.

[1417] A "virtual reality space" is a three-dimensional interface space created using virtual reality technology that users can experience visually and aurally.

[1418] "Eye tracking" is a technology that captures and analyzes a user's eye movements in real time.

[1419] "Gesture recognition" is a technology that captures a user's hand and body movements and analyzes those movements.

[1420] "Voice input" is a technology that acquires and analyzes the user's voice data.

[1421] "Emotional data" is data that indicates the emotional state of a user, obtained from the user's facial expression, tone of voice, etc.

[1422] An "evolved artwork" is a work that has undergone some changes and adjustments to the original artwork based on user reaction and emotional data.

[1423] "Feedback" refers to information such as ratings and comments that users enter about artworks.

[1424] The "learning database" is a database that accumulates past data and feedback that the generative AI will use the next time it generates an artwork.

[1425] "Real time" is a time concept that refers to a system's immediate response and processing without delay.

[1426] This invention provides a system for providing an interactive virtual reality space in which artwork is generated and evolved based on the user's preferences, reactions, and emotions. Specific embodiments of this system are described below.

[1427] When a user logs into the system, the server retrieves profile data. This profile data includes the user's name, age, art preferences, and past reaction data. The device detects whether the user is wearing a smartphone or head-mounted display (HMD) and configures the environment. This includes calibrating the VR equipment and configuring the user's position in the environment. The user is required to enter basic profile information when accessing the system for the first time.

[1428] The server analyzes the acquired profile data and uses a generative AI to analyze the user's art preferences. This process uses machine learning algorithms to identify trends in the user's preferred art styles (abstract, figurative, color patterns, etc.). The generative AI generates an initial artwork based on the analysis results, and the data for this generated artwork is converted into a virtual reality-compatible format.

[1429] The device creates a virtual reality space based on the artwork data sent from the server, providing a three-dimensional interface that responds to the user's visual and auditory senses. Furthermore, the device captures the user's actions in the virtual reality space (eye tracking, gesture recognition, voice input, emotional data acquisition, etc.) in real time and sends the data to the server.

[1430] The server analyzes the user's reaction and emotional data. Based on the analyzed data, the generation AI evolves the artwork. Specifically, it makes changes such as emphasizing parts that the user particularly finds interesting. The data for the evolved artwork is then sent back to the device and instantly reflected in the virtual reality space. In this way, when a new artwork is generated, the existing artwork is updated in real time, completely renewing the user's experience.

[1431] The server collects user feedback and ratings and stores them in the generative AI's learning database. This feedback data is used to optimize the next exhibition. After the exhibition ends, the terminal displays a feedback form to the user, providing an interface where they can enter ratings and comments. The emotion engine obtains emotional data from the user's facial expressions and voice in real time. This includes analyzing facial expressions from the user's camera footage and voice tone from microphone input, and analyzes the obtained emotional data to further adjust the artwork based on their emotional state.

[1432] As a concrete example, consider a scenario where a user generates an artwork using a prompt such as, "Create a colorful abstract painting. In the past, the user has preferred blue and warm color patterns." The artwork generated based on this prompt evolves based on the user's response, e.g., emphasizing the area where the user's gaze is directed or changing color to reflect emotions. As a result, the user can enjoy a more personalized art experience.

[1433] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1434] Step 1:

[1435] A user logs into the system and enters profile data.

[1436] Input: User's name, age, art preferences, and past response data.

[1437] Output: User profile data.

[1438] Specific operation: A user accesses the system using a smartphone or head-mounted display (HMD) and enters profile information when logging in for the first time.

[1439] Step 2:

[1440] The server retrieves the user's profile data and uses generative AI to analyze the user's art preferences.

[1441] Input: The user's profile data obtained in step 1.

[1442] Output: Analysis of user's art preferences.

[1443] What it does: The server analyzes your profile data and uses machine learning algorithms to identify trends in your preferred art styles (abstract, figurative, color patterns, etc.).

[1444] Step 3:

[1445] Generative AI generates an initial artwork based on the analysis and converts it into a VR-compatible format.

[1446] Input: The analysis results from step 2.

[1447] Output: Virtual reality compatible artwork data.

[1448] Specific operation: The generation AI generates a prompt (e.g., "Generate a colorful abstract painting. Users have previously preferred blue and warm color patterns") and generates an artwork based on it. The generated data is converted into a VR format.

[1449] Step 4:

[1450] The device creates a virtual reality space based on the data of the artwork sent from the server.

[1451] Input: Virtual reality-ready artwork data generated in step 3.

[1452] Output: A work of art displayed in a virtual reality space.

[1453] How it works: When a user wears a head-mounted display (HMD), the device displays the artwork, providing the user with a three-dimensional interface that supports both visual and auditory senses.

[1454] Step 5:

[1455] The device captures the user's gaze tracking, gesture recognition, voice input, and emotional data in real time and transmits it to the server.

[1456] Input: User gaze, gesture, voice, and emotion data.

[1457] Output: User response data captured in real time.

[1458] How it works: The device uses the camera and microphone to analyze the user's facial expressions and tone of voice, capture eye tracking and gesture recognition data, and transmit this data to a server in real time.

[1459] Step 6:

[1460] The server analyzes user reaction data and uses generative AI to evolve the artwork.

[1461] Input: User response data obtained in step 5.

[1462] Output: Evolved artwork data.

[1463] How it works: The server analyzes the areas of interest and changes in emotions of the user, and the generative AI modifies the artwork based on that data, for example by emphasizing certain colors or shapes.

[1464] Step 7:

[1465] The device updates and reflects the evolved artwork in the virtual reality space.

[1466] Input: The evolved artwork data generated in step 6.

[1467] Output: An updated artwork in a virtual reality space.

[1468] What it does: The device updates the artwork already displayed in real time, providing the user with a new art experience.

[1469] Step 8:

[1470] The server collects user feedback and ratings and stores them in the generative AI's learning database.

[1471] Input: User feedback and ratings.

[1472] Output: Feedback data stored in the training database.

[1473] Specific operation: After the exhibition ends, the terminal displays a feedback form to the user. The user's ratings and comments are collected, and the server provides this to the generation AI as learning data.

[1474] Step 9:

[1475] The terminal obtains the user's emotional data in real time and further adjusts the artwork based on the user's emotional state.

[1476] Input: User facial and vocal emotion data.

[1477] Output: A coordinated piece of art.

[1478] How it works: The device analyzes the user's emotions based on data obtained from camera footage and microphone input, and then enhances or modifies the artwork based on that.

[1479] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1480] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1481] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1482] [Fourth embodiment]

[1483] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1484] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1485] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1486] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1487] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1488] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1489] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1490] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1491] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1492] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1493] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1494] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1495] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1496] This invention relates to a system for providing an interactive virtual reality (VR) space in which artwork is generated and evolved according to the user's preferences and reactions. Specific aspects of the system are described below.

[1497] User Awareness and Initial Setup

[1498] Server: When a user logs into the system, the server retrieves the user's profile data, which includes the user's name, age, art preferences, past reaction data, etc.

[1499] Device: Detects when the user puts on a VR headset and prepares the VR environment, including calibrating the VR equipment and configuring it to understand the user's position in the environment.

[1500] User: A user puts on a VR headset and controllers and begins accessing the system. First-time users are prompted to enter basic profile information.

[1501] Analysis of user preferences

[1502] Server: Analyzes user profile data and uses generative AI to analyze the user's art preferences. This process uses machine learning algorithms to identify trends in the user's preferred art styles (abstract, figurative, color patterns, etc.).

[1503] Server: The generative AI generates the initial artwork based on the analysis results. The data of this generated artwork is converted into a VR-compatible format.

[1504] Interactive VR space generation

[1505] Terminal: A VR space is constructed based on the artwork data sent from the server, providing a 3D interface that responds to the user's visual and auditory senses.

[1506] Device: Captures the user's actions in the VR space (eye tracking, gesture recognition, voice input, etc.) in real time and sends the data to the server.

[1507] Real-time reactions and artistic evolution

[1508] Server: The generative AI analyzes user reaction data and uses it to evolve the artwork, highlighting areas that the user particularly found interesting.

[1509] Server: Data of the evolved artwork is sent to the device, and is instantly reflected in the VR space.

[1510] Device: As new artworks are generated, existing artworks are updated in real time, completely transforming the user experience.

[1511] Feedback accumulation and optimization

[1512] Server: Collects user feedback and ratings and stores them in the generative AI's learning database. This feedback data is used to optimize the next exhibit.

[1513] Terminal: After the exhibition ends, a feedback form will be displayed to the user, providing an interface where they can enter their ratings and comments.

[1514] Specific examples

[1515] Example 1: When a new user accesses the system and enters their profile information, the generative AI generates an abstract painting in the VR space based on the user's preferences. If the user focuses their gaze on a particular color or shape, the artwork evolves to emphasize that aspect.

[1516] Example 2: When a repeat user revisits the system, artworks are generated that are more tailored to the user based on their past data, resulting in a more refined and personalized art experience for the user.

[1517] As described above, this system combines generative AI and VR technology to create dynamic art exhibits that respond to the user's preferences and reactions. By utilizing digital space, it can provide new art experiences that transcend physical constraints, and can also make a significant contribution to the fields of education and culture.

[1518] The processing flow will be explained below.

[1519] Step 1:

[1520] User: Accesses and logs into the system.

[1521] Specific operation: The user enters their ID and password and presses the login button.

[1522] Step 2:

[1523] Server: Retrieves the user's profile data.

[1524] Specific operation: Search and extract the user's past art viewing history, preferences, rating data, etc. from the database.

[1525] Step 3:

[1526] Device: Detects when the user is wearing a VR headset and configures the environment.

[1527] Specific operation: Calibrate the VR equipment and obtain the user's body position information.

[1528] Step 4:

[1529] Server: Based on the acquired profile data, the server uses generative AI to analyze the user's artistic preferences.

[1530] What it does: It feeds profile data into machine learning algorithms to identify user preferences.

[1531] Step 5:

[1532] Server: Generative AI generates an initial artwork based on the analysis results.

[1533] What it does: The AI ​​model generates digital art by combining appropriate art styles and elements, then converts the data into a VR-compatible format.

[1534] Step 6:

[1535] Server: Sends the generated artwork data to the device.

[1536] Specific operation: Digital art data is compressed and sent to the terminal using a high-speed communication protocol.

[1537] Step 7:

[1538] Terminal: Generates a VR space based on the received art data.

[1539] Specific operation: Using a 3D rendering engine, artwork is displayed in a VR space.

[1540] Step 8:

[1541] User: Experience the artwork in a VR space.

[1542] Specific actions: Users interact with the artwork using eye tracking and gestures.

[1543] Step 9:

[1544] Device: Sends real-time reaction data from the user's gaze tracking and gesture recognition to the server.

[1545] Specific operation: Data acquired by the gaze tracking sensor and gesture recognition system is analyzed and sent to the server in real time.

[1546] Step 10:

[1547] Server: Analyzes user reaction data, and the generative AI uses this data to evolve the artwork.

[1548] Specific behavior: Based on real-time data, the system makes changes to the artwork, such as adding new elements or emphasizing parts that users like.

[1549] Step 11:

[1550] Server: The data of the evolved artwork is sent back to the device.

[1551] Specific behavior: Sends updated art data at high speed.

[1552] Step 12:

[1553] Device: Evolved artworks are reflected in the VR space in real time.

[1554] What it does: Instantly update artwork in the VR space and display new content to users.

[1555] Step 13:

[1556] User: After completing the exhibition experience, enter your evaluation in the feedback form.

[1557] Specific action: Fill out the feedback form that appears and provide your rating and comments about the art experience, then submit it.

[1558] Step 14:

[1559] Server: Stores user feedback data and provides it to the generation AI for next display optimization.

[1560] Specific operation: Feedback data is accumulated in a database and used as learning data for the generative AI.

[1561] Example 1

[1562] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1563] Conventional art exhibition systems have struggled to dynamically generate and evolve artworks based on individual user preferences and real-time responses. They also have limited means to effectively incorporate interactive data, such as user gaze tracking and gesture recognition. As a result, the experience for each user is not fully personalized, and there is no appropriate mechanism for incorporating user feedback into future exhibitions.

[1564] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1565] In this invention, the server includes means for acquiring user profile data, means for using artificial intelligence to analyze user preferences based on the acquired user profile data, means for rendering a digital artwork generated by the artificial intelligence in a virtual reality space, means for acquiring data based on the user's gaze tracking and motion recognition in real time, means for analyzing the acquired data and using artificial intelligence to evolve the digital artwork, and means for updating and reflecting the evolved digital artwork in the virtual reality space, thereby enabling a dynamic and personalized art experience that reflects individual user preferences and real-time reactions.

[1566] "User profile data" refers to any information related to a user who logs into the system, such as the user's name, age, art preferences, and past response data.

[1567] "Artificial intelligence" refers to technologies including machine learning algorithms and deep learning models that analyze user profile data and real-time reaction data to generate and evolve appropriate artworks.

[1568] "Digital artwork" refers to art generated by artificial intelligence, the data of which is converted into a format (e.g., FBX or OBJ) that can be rendered in a virtual reality space.

[1569] "Virtual reality space" refers to a three-dimensional digital environment that users can immerse themselves in using devices such as VR headsets.

[1570] "Eye tracking" refers to a technology that tracks the direction a user is looking or a specific object in a virtual reality space in real time.

[1571] "Motion recognition" refers to the technology of capturing a user's gestures and body movements with sensors and analyzing them.

[1572] "Real-time data" refers to data obtained through user gaze tracking and / or motion recognition that is processed and analyzed in real time.

[1573] MODE FOR CARRYING OUT THE INVENTION

[1574] This invention relates to a system that provides an interactive virtual reality (VR) space in which digital artwork is generated and evolved according to the preferences and reactions of users. Specific embodiments will be described below.

[1575] User Awareness and Initial Setup

[1576] Server: When a user logs into the system, the server retrieves the user's profile data (such as name, age, art preferences, past reaction data, etc.) This profile data is retrieved using database queries, and the data is cleansed and analyzed using Python libraries (e.g., Pandas, NumPy).

[1577] Device: When a user puts on a VR headset (e.g., a general-purpose VR headset), the device detects this information and prepares the VR environment, including calibrating the VR device and configuring it to understand the user's position (e.g., 6DOF tracking).

[1578] User: The user puts on the VR headset and controllers and begins accessing the system. First-time users are asked to enter basic profile information (such as name, age, and art preferences) through a form in the interface.

[1579] Analysis of user preferences

[1580] Server: The server analyzes the acquired user profile data and uses generative AI to analyze the user's art preferences. In this process, machine learning algorithms (e.g., TensorFlow, PyTorch) are used to identify trends in the user's preferred art styles (abstract art, figurative art, color patterns, etc.).

[1581] Server: Generative AI (e.g., GPT-3, GANs) generates the initial digital artwork based on the analysis results. The Assimp library is used to convert this generated artwork data into a VR-compatible format (e.g., FBX format or OBJ format).

[1582] Interactive VR space generation

[1583] Terminal: Using Unity or Unreal Engine, a VR space is created based on the artwork data sent from the server. This provides a 3D interface that responds to the user's vision and hearing.

[1584] Device: Configure it using the necessary APIs (e.g., OpenVR SDK, general-purpose VR SDK) to capture the user's actions in the VR space (e.g., eye tracking, motion recognition, voice input, etc.) in real time.

[1585] Real-time reactions and artistic evolution

[1586] Server: Analyzes the user's real-time reaction data (eye tracking data, motion recognition data, etc.), and the generative AI evolves the artwork based on this. Specifically, computer vision technology (e.g., OpenCV) is used to analyze the gaze data.

[1587] Server: Sends the evolved artwork data to the device. This process uses WebSocket to exchange data in real time.

[1588] Device: When a new artwork is generated, the existing artwork is instantly updated in the VR space, leveraging the real-time rendering capabilities of Unity and Unreal Engine, so the changes are instantly reflected in the user's vision.

[1589] Feedback accumulation and optimization

[1590] Server: After the exhibition, the feedback and evaluations collected from users are stored in the AI's learning database. This data will be used for future art generation.

[1591] Terminal: After the exhibition ends, a feedback form is displayed to the user, providing an interface where they can enter their ratings and comments. This form data is collected using HTML and JavaScript and sent to the server.

[1592] Specific examples

[1593] Example 1:

[1594] Server: When a new user logs in, it collects profile information and queries the database. Python is used to clean the data and extract preferences. The generative AI generates abstract art based on the data and converts it into FBX format.

[1595] Device: Receives artwork data and displays it in the VR space using Unity. When the user directs their gaze at a specific color or shape, eye-tracking data is sent to the server.

[1596] Server: To evolve the artwork based on the eye-tracking data, the generative AI updates the artwork design. It converts the new data into a VR format and sends it to the device.

[1597] Device: The new artwork is instantly updated in the VR space, allowing users to experience the changes in real time.

[1598] Example 2:

[1599] Server: When a repeat user accesses the site again, the initial settings are made based on the user's past profile data and responses, and an art piece is generated that is further tailored to the user.

[1600] Device: Compares existing art with new art and dynamically updates the VR space. When the user responds with gestures to specific shapes, the information is sent to the server in real time.

[1601] Server: The artwork is partially evolved based on gesture recognition data and the information is sent to the device.

[1602] On your device: Display updated artwork so that your art experience is more refined and personalized than the last time.

[1603] Prompt Sentence Examples

[1604] "Describe the process of generating abstract paintings based on user profile information and analyzing the user's eye-tracking data to evolve the artwork."

[1605] As described above, this system combines generative AI and VR technology to provide a dynamic art experience that responds to the user's preferences and reactions. By utilizing digital space, it can provide new art experiences that transcend physical constraints and make significant contributions to the fields of education and culture.

[1606] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1607] Program processing flow

[1608] Step 1: User Awareness and Initial Setup

[1609] Server: When a user logs in to the system, the server retrieves the user's profile data (such as name, age, art preferences, and past reaction data) using a database query and cleanses the data using Python libraries (e.g., Pandas and NumPy).

[1610] Input: User login information.

[1611] Output: Cleansed user profile data.

[1612] Device: When the user puts on the VR headset, the device detects this information and prepares the VR environment. Specifically, it calibrates the VR equipment and sets up the user's position (for example, 6DOF tracking).

[1613] Input: VR headset wearing information.

[1614] Output: Calibrated VR environment setup.

[1615] User: The user puts on the VR headset and controllers and begins accessing the system. First-time users enter basic profile information (such as name, age, and art preferences) through a form in the interface.

[1616] Input: Basic profile information.

[1617] Output: The entered profile information.

[1618] Step 2: Analyzing user preferences

[1619] Server: The server analyzes user profile data and uses generative AI (e.g., TensorFlow or PyTorch) to analyze the user's art preferences. It uses machine learning algorithms to identify trends in the user's preferred art styles (e.g., abstract, figurative, color patterns, etc.).

[1620] Input: Cleansed user profile data.

[1621] Output: User's art preference information.

[1622] Server: The generative AI generates the initial digital artwork based on the analysis results. The generated artwork data is converted into a VR-compatible format (e.g., FBX or OBJ format) using the Assimp library.

[1623] Input: User's art preference information.

[1624] Output: Digital artwork data in a VR-compatible format.

[1625] Step 3: Creating an interactive VR space

[1626] Device: Using Unity or Unreal Engine, a VR space is created based on the artwork data sent from the server, providing a 3D interface that responds to the user's visual and auditory senses.

[1627] Input: Digital artwork data in a VR-compatible format.

[1628] Output: The constructed VR space.

[1629] Device: Captures the user's actions in the VR space (eye tracking, motion recognition, voice input, etc.) in real time and sends the data to the server. Uses the required API (e.g., OpenVR SDK, general-purpose VR SDK).

[1630] Input: User operation data.

[1631] Output: Operation data sent to the server.

[1632] Step 4: Real-time reactions and artistic evolution

[1633] Server: Analyzes the user's real-time reaction data (eye tracking data, motion recognition data, etc.), and the generative AI evolves the artwork based on this data. Computer vision technology (e.g., OpenCV) is used to analyze the gaze data.

[1634] Input: Real-time user response data.

[1635] Output: Evolved digital artwork data.

[1636] Server: Sends the evolved artwork data to the device. Data is exchanged in real time using WebSocket.

[1637] Input: Evolved digital artwork data.

[1638] Output: Artwork data sent to the device.

[1639] Device: As new artworks are generated, existing artworks are updated in real time within the VR space, utilizing the real-time rendering capabilities of Unity and Unreal Engine.

[1640] Input: Evolved digital artwork data.

[1641] Output: Updated VR space.

[1642] Step 5: Gather feedback and optimize

[1643] Server: After the exhibition is over, the feedback and evaluations collected from users are stored in the Generative AI's learning database. This data is used to optimize the next exhibition.

[1644] Input: User feedback data.

[1645] Output: Saved feedback data.

[1646] Terminal: After the exhibition ends, a feedback form is displayed to the user, providing an interface where they can enter their ratings and comments. The form data is collected using HTML and JavaScript and sent to the server.

[1647] Input: User ratings and comments.

[1648] Output: The feedback form data sent to the server.

[1649] Through these processing steps, the system provides a dynamic art experience that responds to user preferences and real-time responses.

[1650] (Application example 1)

[1651] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1652] Conventional entertainment systems for autonomous vehicles have the drawback of being difficult to provide content that responds to individual user preferences and real-time responses, and are difficult to provide an experience with enhanced visual and tactile interactivity due to the limited environment inside the vehicle.

[1653] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1654] In this invention, the server includes means for acquiring user profile data, means for using a generation AI to analyze the user's art preferences based on the acquired profile data, means for rendering artwork generated by the generation AI in a virtual reality space, means for acquiring data based on the user's gaze tracking and gesture recognition in real time, means for analyzing the acquired data and using the generation AI to evolve the artwork, means for updating and reflecting the evolved artwork in the virtual reality space, and means for providing a terminal for using these means within an autonomous vehicle, thereby enabling the provision of a high-quality, personalized art experience in real time even within an autonomous vehicle.

[1655] "User Profile Data" means data that includes personal information about you, such as your name, age, and artistic preferences.

[1656] "Generative AI" is artificial intelligence that uses machine learning algorithms to generate and evolve artworks based on user preferences.

[1657] A "virtual reality space" is a three-dimensional virtual space that provides users with an immersive experience through visual and auditory interfaces.

[1658] "Eye tracking" is a technology that detects and analyzes the user's eye movements in real time.

[1659] "Gesture recognition" is a technology that uses cameras and sensors to detect and analyze the movements of a user's hands and fingers.

[1660] "Real-time acquisition means" refers to a system or device that instantly collects interaction data such as user gaze and gestures.

[1661] An "autonomous vehicle" is a vehicle that drives autonomously using artificial intelligence and sensor technology.

[1662] A "terminal" is a device or system (e.g., a head-mounted display) that allows a user to access a virtual reality space.

[1663] A "feedback form" is an interface that allows users to enter their opinions and evaluations of artworks.

[1664] The "generative AI learning database" is a database used to accumulate user feedback and reaction data and improve the performance of the generative AI.

[1665] The embodiments of the present invention will be specifically described below.

[1666] Retrieving a user profile

[1667] When a user logs into the system, the server retrieves the user's profile data, which includes the user's name, age, art preferences, past interaction data, etc. This provides the basis for collecting data to provide a personalized experience for each user.

[1668] User Awareness and Initial Setup

[1669] The device detects when the user is wearing a VR headset and prepares the VR environment, including calibrating the head-mounted display (e.g., Oculus Quest) and configuring it to understand the user's position. When a user first accesses the system, they are required to enter basic profile information.

[1670] Art taste analysis

[1671] The server analyzes the profile data and uses generative AI to analyze the user's art preferences. This process uses machine learning algorithms to identify trends in the user's preferred art styles (abstract, figurative, color patterns, etc.).

[1672] Generating and Rendering Initial Artwork

[1673] The generative AI generates an initial artwork based on the user's preferences. The data for this artwork is converted into a format compatible with the virtual reality space and sent to the device. The device then creates a virtual reality space based on the data sent from the server. This provides a 3D interface that responds to the user's visual and auditory senses.

[1674] Interactive operation and real-time updates

[1675] The device captures the user's actions in the virtual reality space (eye tracking, gesture recognition, voice input, etc.) in real time and sends the data to the server. The server analyzes the user's reaction data, and the generative AI uses this data to evolve the artwork. Changes are made, such as emphasizing areas that the user has shown particular interest in. Data on the evolved artwork is sent to the device and instantly reflected in the virtual reality space, completely transforming the user's experience.

[1676] Feedback collection and optimization

[1677] The server collects user feedback and ratings and stores them in the generative AI's learning database. This feedback data is used to optimize the next exhibit. After the exhibit is over, the terminal displays a feedback form to the user, providing an interface where they can enter ratings and comments.

[1678] Use in self-driving vehicles

[1679] This system is installed in autonomous vehicles, allowing users to enjoy art experiences using VR headsets while traveling. The in-car device captures the user's actions in real time and works with a server to evolve the artwork, providing immersive entertainment even while traveling.

[1680] Examples:

[1681] Example 1: A new user accesses the system and enters their profile information. Generative AI generates an abstract painting in a virtual reality space based on the user's preferences. If the user focuses their gaze on a particular color or shape, the artwork evolves to emphasize that feature.

[1682] Example 2: When a returning user revisits the system, the system generates more personalized artwork based on their past data, resulting in a more refined and personalized art experience.

[1683] Example prompt sentence:

[1684] "Develop an app for autonomous vehicles that allows passengers to use VR headsets to experience and evolve art according to their preferences."

[1685] In this way, this invention combines generative AI and virtual reality technology to create dynamic art exhibits that respond to user preferences and reactions, providing a new entertainment experience even inside autonomous vehicles.

[1686] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1687] Step 1:

[1688] When a user logs in to the system, the server retrieves the user's profile data. The profile data includes the user's name, age, art preferences, and past response data. This provides the basis for the user to receive a personalized experience. The input is the user's login information, and the output is the user's profile data.

[1689] Step 2:

[1690] The device detects when the user puts on a VR headset and prepares the virtual reality environment. This includes calibrating the head-mounted display (e.g., Oculus Quest) and determining the user's position. The input is the state of the VR headset and user position information, and the output is information that the virtual reality environment is ready.

[1691] Step 3:

[1692] The server analyzes the acquired profile data and uses generative AI to analyze the user's art preferences. Machine learning algorithms are used to identify trends in the user's preferred art styles (abstract, figurative, color patterns, etc.). The input is the profile data, and the output is the analysis of the user's art preferences.

[1693] Step 4:

[1694] The server uses a generative AI to generate an initial artwork based on the user's preferences, converts it into a format compatible with the virtual reality space, and sends it to the device. The input is the user's art preference analysis results, and the output is VR-compatible initial artwork data.

[1695] Step 5:

[1696] The device constructs a virtual reality space based on the artwork data received from the server, providing a 3D interface that responds to the user's visual and auditory senses. The input is the initial artwork data, and the output is the constructed virtual reality space.

[1697] Step 6:

[1698] The device captures the user's actions in the virtual reality space (eye tracking, gesture recognition, voice input) in real time and sends the data to the server. The input is the user's action data, and the output is the data sent to the server.

[1699] Step 7:

[1700] The server analyzes the user's operation data, and the generative AI evolves the artwork based on this data. In particular, it makes changes such as emphasizing the parts that the user showed interest in. The input is the user's operation data, and the output is the evolved artwork data.

[1701] Step 8:

[1702] The server sends the evolved artwork data to the device, which then instantly reflects it in the virtual reality space. The input is the evolved artwork data, and the output is the updated virtual reality space.

[1703] Step 9:

[1704] The server collects user feedback and ratings and stores them in the generative AI's learning database. This feedback data is used to optimize the next display. The input is user feedback, and the output is an updated learning database.

[1705] Step 10:

[1706] The terminal displays a feedback form to the user after the exhibition has ended, providing an interface where users can enter their ratings and comments. The input triggers the end of the exhibition, and the output is the feedback information from the user.

[1707] The entire process enables a personalized art experience to be delivered in real time within the autonomous vehicle, and the system dynamically adapts to the user's preferences and reactions, delivering high-quality, immersive entertainment.

[1708] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1709] This invention relates to a system for providing an interactive virtual reality (VR) space that generates and evolves artwork by recognizing the user's preferences, reactions, and emotions. Specific aspects of the system are described below.

[1710] User Awareness and Initial Setup

[1711] Server: When a user logs into the system, the server retrieves the user's profile data, which includes the user's name, age, art preferences, past reaction data, etc.

[1712] Device: Detects when the user puts on a VR headset and configures the environment, including calibrating the VR equipment and locating the user in the environment.

[1713] User: A user puts on a VR headset and controllers and begins accessing the system. First-time users are prompted to enter basic profile information.

[1714] Analysis of user preferences

[1715] Server: Analyzes user profile data and uses generative AI to analyze the user's art preferences. This process uses machine learning algorithms to identify trends in the user's preferred art styles (abstract, figurative, color patterns, etc.).

[1716] Server: The generative AI generates the initial artwork based on the analysis results. The data of this generated artwork is converted into a VR-compatible format.

[1717] Interactive VR space generation

[1718] Terminal: A VR space is constructed based on the artwork data sent from the server, providing a 3D interface that responds to the user's visual and auditory senses.

[1719] Terminal: Captures the user's actions in the VR space (eye tracking, gesture recognition, voice input, emotion recognition using the emotion engine, etc.) in real time and sends the data to the server.

[1720] Real-time reactions and artistic evolution

[1721] Server: The generative AI analyzes the user's reaction and emotional data and uses this data to evolve the artwork, highlighting areas that the user particularly found interesting.

[1722] Server: Data of the evolved artwork is sent to the device, and is instantly reflected in the VR space.

[1723] Device: As new artworks are generated, existing artworks are updated in real time, completely transforming the user experience.

[1724] Feedback accumulation and optimization

[1725] Server: Collects user feedback and ratings and stores them in the generative AI's learning database. This feedback data is used to optimize the next exhibit.

[1726] Terminal: After the exhibition ends, a feedback form will be displayed to the user, providing an interface where they can enter their ratings and comments.

[1727] Further processing for emotion recognition

[1728] On the device: The emotion engine captures emotional data from the user's facial expressions and voice in real time, including analyzing facial expressions from the user's camera footage and tone of voice from the microphone input.

[1729] Server: Analyzes the acquired emotional data and further adjusts the artwork based on the user's emotional state, such as further emphasizing parts that made the user feel surprised or happy.

[1730] Specific examples

[1731] Example 1: When a new user accesses the system and enters their profile information, the generative AI generates an abstract painting in the VR space based on the user's preferences. If the user focuses their gaze on a particular color or shape, and the emotion engine detects the user's joy, the artwork evolves to emphasize that part.

[1732] Example 2: When a repeat user revisits the system, a more personalized artwork is generated based on their past and emotional data, resulting in a more refined and personalized art experience for the user.

[1733] Summary of implementation procedures

[1734] The system combines generative AI, VR technology, eye tracking, gesture recognition, voice input, and emotion recognition to create dynamic and personalized art exhibits that respond to the user's preferences, reactions, and emotions. By utilizing digital space, it can provide new art experiences that transcend physical constraints, and make significant contributions to the fields of education and culture.

[1735] The processing flow will be explained below.

[1736] Step 1:

[1737] User: Accesses and logs into the system.

[1738] Specific operation: The user enters their ID and password and presses the login button.

[1739] Step 2:

[1740] Server: Retrieves the user's profile data.

[1741] Specific operation: Search and extract the user's past art viewing history, preferences, rating data, etc. from the database.

[1742] Step 3:

[1743] Device: Detects when the user is wearing a VR headset and configures the environment.

[1744] Specific operation: Calibrate the VR equipment and obtain the user's body position information.

[1745] Step 4:

[1746] Server: Based on the acquired profile data, the server uses generative AI to analyze the user's artistic preferences.

[1747] What it does: It feeds profile data into machine learning algorithms to identify user preferences.

[1748] Step 5:

[1749] Server: Generative AI generates an initial artwork based on the analysis results.

[1750] What it does: Generative AI generates digital art by combining appropriate art styles and elements, then converts the data into a VR-compatible format.

[1751] Step 6:

[1752] Server: Sends the generated artwork data to the device.

[1753] Specific operation: Digital art data is compressed and sent to the terminal using a high-speed communication protocol.

[1754] Step 7:

[1755] Terminal: Generates a VR space based on the received art data.

[1756] Specific operation: Using a 3D rendering engine, artwork is displayed in a VR space.

[1757] Step 8:

[1758] User: Experience the artwork in a VR space.

[1759] Specific Actions: Users interact with the artwork using eye tracking, gestures, voice input, and emotion recognition.

[1760] Step 9:

[1761] Device: Sends real-time reaction data from the user's gaze tracking, gesture recognition, voice input, and emotion recognition to the server.

[1762] Specific operation: All data acquired by the gaze tracking sensor, gesture recognition system, microphone, and emotion engine (facial expression analysis, tone of voice analysis) is analyzed and sent to the server in real time.

[1763] Step 10:

[1764] Server: Analyzes user reaction and emotional data, and the generative AI uses this data to evolve the artwork.

[1765] Specific behavior: Based on real-time data, the artwork is modified to add new elements or highlight parts that users like or that elicit an emotional response.

[1766] Step 11:

[1767] Server: The data of the evolved artwork is sent back to the device.

[1768] Specific behavior: Sends updated art data at high speed.

[1769] Step 12:

[1770] Device: Evolved artworks are reflected in the VR space in real time.

[1771] What it does: Instantly update artwork in the VR space and display new content to users.

[1772] Step 13:

[1773] User: After completing the exhibition experience, enter your evaluation in the feedback form.

[1774] Specific action: Fill out the feedback form that appears and provide your rating and comments about the art experience, then submit it.

[1775] Step 14:

[1776] Server: Stores user feedback data and provides it to the generation AI for next display optimization.

[1777] Specific operation: Feedback data is accumulated in a database and used as learning data for the generative AI.

[1778] Example 2

[1779] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1780] Conventional art exhibition systems have difficulty reflecting user preferences, reactions, and emotions in real time, making it difficult to provide a personalized art experience. Furthermore, there is a lack of mechanisms for incorporating user feedback into future exhibitions, making it difficult to provide a continuously optimized experience. To solve these issues, a system is needed that can acquire user profile data, reaction data, and emotional data in real time and evolve artworks based on that data.

[1781] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1782] In this invention, the server includes means for acquiring user attribute information, means for using a generation AI to analyze the user's artistic preferences based on the acquired attribute information, means for rendering artwork generated by the generation AI in a virtual reality space, means for acquiring data based on the user's gaze tracking and motion recognition in real time, means for analyzing the acquired data and using the generation AI to evolve the artwork, means for updating and reflecting the evolved artwork in the virtual reality space, and means for acquiring the user's emotional state in real time and analyzing the data, thereby enabling dynamic and personalized art exhibits based on the user's preferences, reactions, and emotions.

[1783] "User demographic information" is information that indicates a user's personal characteristics and preferences, such as the user's name, age, hobbies, and past activity.

[1784] "Generative AI" is an artificial intelligence technology used to generate or evolve artworks based on user attribute information and reaction data.

[1785] A "virtual reality space" is a three-dimensional digital environment that users access through a VR headset.

[1786] "Eye tracking" is a technology that detects a user's eye movements and gaze position in real time.

[1787] "Motion recognition" is a technology that detects a user's hand movements and body gestures and captures them as data.

[1788] "Real-time acquisition" means that data is collected almost simultaneously and processed immediately.

[1789] "Analysis" is the process of examining acquired data in detail and extracting meaning and patterns.

[1790] "Evolution" is the process of dynamically changing or improving a work of art based on user reactions and emotions.

[1791] "Emotional state" refers to the emotions such as joy, surprise, or anger that a user is feeling at a particular moment.

[1792] An "evaluation form" is an interface that allows users to enter their opinions and thoughts about artworks.

[1793] These definitions clearly describe each element of the system.

[1794] The present invention relates to a system for providing an interactive virtual reality space in which artworks are generated and evolved by recognizing the preferences, reactions, and emotions of users. Specific embodiments will be described below.

[1795] User Awareness and Initial Setup

[1796] server:

[1797] When a user logs in to the system, the server obtains the user's attribute information. This attribute information includes the user's name, age, artistic preferences, past reaction data, etc. For hardware, a general server or cloud infrastructure is used. For specific software, a database management system (e.g., MySQL, PostgreSQL) is used.

[1798] Device:

[1799] It detects when a user puts on a virtual reality (VR) headset and configures the VR environment, including calibrating the VR device (e.g., Oculus Quest 2) and setting up the user's position. The device configures the environment using dedicated VR software (e.g., Unity, Unreal Engine).

[1800] user:

[1801] Users put on a VR headset and controllers to begin accessing the system, and new users are asked to enter basic demographic information such as their name, age, and preferences.

[1802] Analysis of user preferences

[1803] server:

[1804] The acquired attribute information is analyzed and a generative AI is used to identify the user's artistic preferences. In this process, a generative AI model (e.g., GPT-3) is used to identify the user's preferred artistic style (abstract, figurative, color patterns, etc.). Specifically, the following prompt sentences are input to the AI ​​model:

[1805] "Generate abstract paintings that users love and incorporate past preferences."

[1806] Generating the initial artwork

[1807] server:

[1808] Based on the analysis results, the generative AI generates an initial artwork, which is then converted into a VR-compatible format (e.g., GLTF format) and sent to the device.

[1809] Building an interactive VR space

[1810] Device:

[1811] The VR space is constructed based on the artwork data sent from the server. In this process, VR platforms such as Unity and Unreal Engine are used to provide a three-dimensional interface that responds to the user's visual and auditory senses.

[1812] Real-time data capture and transmission

[1813] Device:

[1814] The system captures user actions (eye tracking, motion recognition, and voice input) in real time and sends the data to a server. For eye tracking, it uses a built-in camera and sensors, and for motion recognition, it uses a remote controller and hand tracking technology. For voice input, it uses a microphone, which is then analyzed by voice recognition software (e.g., Google Cloud Speech-to-Text).

[1815] Device: Using an emotion engine (e.g., Emotion API), emotional data is acquired from the user's facial expressions and voice and sent to the server. Specifically, facial expressions are analyzed from the user's camera footage, and the tone of voice is analyzed from microphone input.

[1816] Real-time reactions and artistic evolution

[1817] server:

[1818] The AI ​​analyzes user reaction and emotional data and uses this data to evolve the artwork. For example, if a user expresses interest in a particular color or shape, it will emphasize that part.

[1819] server:

[1820] By sending data of the evolved artwork to the device, it is instantly reflected in the VR space.

[1821] Device:

[1822] As new artworks are generated, existing works are updated in real time, providing a fresh user experience.

[1823] Feedback collection and optimization

[1824] server:

[1825] User feedback and ratings are collected and stored in the generative AI's learning database, which will further optimize the next art exhibit.

[1826] Device:

[1827] After the exhibition ends, a feedback form will be displayed to users, providing an interface where they can enter their ratings and comments.

[1828] Further processing for emotion recognition

[1829] Device:

[1830] The emotion engine captures emotional data from the user's facial expressions and voice in real time, including analyzing facial expressions from the user's camera footage and tone of voice from microphone input.

[1831] server:

[1832] The acquired emotional data is analyzed and the artwork is further adjusted based on the user's emotional state, for example by further emphasizing parts that made the user feel surprised or happy.

[1833] This system will enable personalized art exhibitions based on the user's preferences, reactions, and emotions, and will also provide a new art experience that transcends physical constraints through the use of digital technology.

[1834] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1835] Step 1:

[1836] When a user logs into the system, the server obtains the user's attribute information (name, age, artistic preferences, and past response data).

[1837] Input: User ID and login information

[1838] Data processing: Search the database using the user ID as a key and obtain the corresponding user information

[1839] Output: Retrieved user attribute information

[1840] Step 2:

[1841] The server analyzes the acquired attribute information and uses generative AI to identify the user's artistic preferences.

[1842] Input: User attribute information

[1843] Data calculation: Input prompts into the generative AI model and perform analysis that reflects user preferences.

[1844] Output: Information about the user's artistic preferences

[1845] Example prompt: "Generate abstract paintings that users will love and incorporate their past preferences."

[1846] Step 3:

[1847] The server uses generative AI to generate an initial artwork based on the user's preferences.

[1848] Input: Information about the user's artistic preferences

[1849] Data Computation: Generative AI model generates initial artwork based on prompt text

[1850] Output: Initial artwork data (converted to VR format)

[1851] Operation: The generated artwork data is converted into VR-compatible GLTF format and sent to the device.

[1852] Step 4:

[1853] The device creates a VR space based on the artwork data sent from the server.

[1854] Input: Initial artwork data in GLTF format

[1855] Data processing: Using VR software such as Unity or Unreal Engine, we create a three-dimensional VR space.

[1856] Output: The VR space experienced by the user

[1857] Action: Calibrate the VR device and set up the device to understand the user's position.

[1858] Step 5:

[1859] The device captures the user's gaze tracking, movement recognition, and voice input in real time within the VR space and sends the data to a server.

[1860] Input: User gaze, movement, and voice data

[1861] Data processing: Capture and process data using eye-tracking sensors, gesture recognition sensors, and voice recognition software

[1862] Output: Captured real-time data

[1863] How it works: Uses the emotion engine to get user emotion data and send it to the server

[1864] Step 6:

[1865] The server analyzes the real-time data it receives, and the generative AI uses this data to evolve the artwork.

[1866] Input: Captured real-time data and sentiment data

[1867] Data Computation: Generative AI models analyze real-time data and dynamically update artwork based on user responses

[1868] Output: Improved artwork data

[1869] How it works: If a particular color or shape shows interest, it highlights that area in near real time.

[1870] Step 7:

[1871] The server transmits data of the evolved artwork to the terminal.

[1872] Input: Evolved artwork data

[1873] Data processing: Reconverting the evolved artwork data into VR format

[1874] Output: Reconverted artwork data

[1875] Operation: Transmitted to the device and reflected instantly in the VR space

[1876] Step 8:

[1877] The device updates the existing VR space in real time as new artworks are created.

[1878] Input: Evolved artwork data

[1879] Data processing: Using VR software such as Unity or Unreal Engine, we update existing VR spaces with new artwork.

[1880] Output: Updated VR space

[1881] What it does: A completely new user experience

[1882] Step 9:

[1883] The server collects user feedback and ratings and stores them in the generative AI's learning database.

[1884] Input: User feedback and rating data

[1885] Data processing: Data collected through the feedback form is saved in the AI ​​learning database.

[1886] Output: Accumulated feedback data

[1887] Action: The display will be further optimized from next time onwards.

[1888] Step 10:

[1889] After the exhibition ends, the terminal displays a feedback form to users, providing an interface where they can enter their ratings and comments.

[1890] Input: User ratings and comments

[1891] Data processing: Evaluation data is collected using a feedback form and sent to the server.

[1892] Output: Collected assessment data

[1893] Behavior: Updates the interface presented to the user to prompt for feedback

[1894] In this way, by clarifying the specific actions, inputs and outputs performed at each step, an interactive VR artwork is provided that evolves in real time in response to the user's preferences, reactions and emotions.

[1895] (Application example 2)

[1896] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1897] Conventional art exhibition systems have struggled to generate personalized artworks that reflect users' preferences, reactions, and emotions in real time, and lacked a system for optimizing the next exhibition using feedback and emotional data, making it difficult to increase user satisfaction.

[1898] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1899] In this invention, the server includes means for acquiring user profile data, means for using a generation AI to analyze the user's art preferences based on the acquired profile data, means for rendering artwork generated by the generation AI in a virtual reality space, means for acquiring the user's gaze tracking, gesture recognition, voice input, and emotional data in real time, means for analyzing the acquired data and using the generation AI to evolve the artwork, means for updating and reflecting the evolved artwork in the virtual reality space, means for enhancing or adjusting the artwork based on the user's emotional data, and means for collecting user feedback and accumulating it in the generation AI's learning database, thereby enabling the provision of personalized artwork that reflects the user's preferences, reactions, and emotions in real time.

[1900] "User Profile Data" means personal information such as your name, age, art preferences, and past interaction data.

[1901] "Generative AI" is artificial intelligence that uses machine learning algorithms to generate and evolve artworks based on user data.

[1902] A "virtual reality space" is a three-dimensional interface space created using virtual reality technology that users can experience visually and aurally.

[1903] "Eye tracking" is a technology that captures and analyzes a user's eye movements in real time.

[1904] "Gesture recognition" is a technology that captures a user's hand and body movements and analyzes those movements.

[1905] "Voice input" is a technology that acquires and analyzes the user's voice data.

[1906] "Emotional data" is data that indicates the emotional state of a user, obtained from the user's facial expression, tone of voice, etc.

[1907] An "evolved artwork" is a work that has undergone some changes and adjustments to the original artwork based on user reaction and emotional data.

[1908] "Feedback" refers to information such as ratings and comments that users enter about artworks.

[1909] The "learning database" is a database that accumulates past data and feedback that the generative AI will use the next time it generates an artwork.

[1910] "Real time" is a time concept that refers to a system's immediate response and processing without delay.

[1911] This invention provides a system for providing an interactive virtual reality space in which artwork is generated and evolved based on the user's preferences, reactions, and emotions. Specific embodiments of this system are described below.

[1912] When a user logs into the system, the server retrieves profile data. This profile data includes the user's name, age, art preferences, and past reaction data. The device detects whether the user is wearing a smartphone or head-mounted display (HMD) and configures the environment. This includes calibrating the VR equipment and configuring the user's position in the environment. The user is required to enter basic profile information when accessing the system for the first time.

[1913] The server analyzes the acquired profile data and uses a generative AI to analyze the user's art preferences. This process uses machine learning algorithms to identify trends in the user's preferred art styles (abstract, figurative, color patterns, etc.). The generative AI generates an initial artwork based on the analysis results, and the data for this generated artwork is converted into a virtual reality-compatible format.

[1914] The device creates a virtual reality space based on the artwork data sent from the server, providing a three-dimensional interface that responds to the user's visual and auditory senses. Furthermore, the device captures the user's actions in the virtual reality space (eye tracking, gesture recognition, voice input, emotional data acquisition, etc.) in real time and sends the data to the server.

[1915] The server analyzes the user's reaction and emotional data. Based on the analyzed data, the generation AI evolves the artwork. Specifically, it makes changes such as emphasizing parts that the user particularly finds interesting. The data for the evolved artwork is then sent back to the device and instantly reflected in the virtual reality space. In this way, when a new artwork is generated, the existing artwork is updated in real time, completely renewing the user's experience.

[1916] The server collects user feedback and ratings and stores them in the generative AI's learning database. This feedback data is used to optimize the next exhibition. After the exhibition ends, the terminal displays a feedback form to the user, providing an interface where they can enter ratings and comments. The emotion engine obtains emotional data from the user's facial expressions and voice in real time. This includes analyzing facial expressions from the user's camera footage and voice tone from microphone input, and analyzes the obtained emotional data to further adjust the artwork based on their emotional state.

[1917] As a concrete example, consider a scenario where a user generates an artwork using a prompt such as, "Create a colorful abstract painting. In the past, the user has preferred blue and warm color patterns." The artwork generated based on this prompt evolves based on the user's response, e.g., emphasizing the area where the user's gaze is directed or changing color to reflect emotions. As a result, the user can enjoy a more personalized art experience.

[1918] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1919] Step 1:

[1920] A user logs into the system and enters profile data.

[1921] Input: User's name, age, art preferences, and past response data.

[1922] Output: User profile data.

[1923] Specific operation: A user accesses the system using a smartphone or head-mounted display (HMD) and enters profile information when logging in for the first time.

[1924] Step 2:

[1925] The server retrieves the user's profile data and uses generative AI to analyze the user's art preferences.

[1926] Input: The user's profile data obtained in step 1.

[1927] Output: Analysis of user's art preferences.

[1928] What it does: The server analyzes your profile data and uses machine learning algorithms to identify trends in your preferred art styles (abstract, figurative, color patterns, etc.).

[1929] Step 3:

[1930] Generative AI generates an initial artwork based on the analysis and converts it into a VR-compatible format.

[1931] Input: The analysis results from step 2.

[1932] Output: Virtual reality compatible artwork data.

[1933] Specific operation: The generation AI generates a prompt (e.g., "Generate a colorful abstract painting. Users have previously preferred blue and warm color patterns") and generates an artwork based on it. The generated data is converted into a VR format.

[1934] Step 4:

[1935] The device creates a virtual reality space based on the data of the artwork sent from the server.

[1936] Input: Virtual reality-ready artwork data generated in step 3.

[1937] Output: A work of art displayed in a virtual reality space.

[1938] How it works: When a user wears a head-mounted display (HMD), the device displays the artwork, providing the user with a three-dimensional interface that supports both visual and auditory senses.

[1939] Step 5:

[1940] The device captures the user's gaze tracking, gesture recognition, voice input, and emotional data in real time and transmits it to the server.

[1941] Input: User gaze, gesture, voice, and emotion data.

[1942] Output: User response data captured in real time.

[1943] How it works: The device uses the camera and microphone to analyze the user's facial expressions and tone of voice, capture eye tracking and gesture recognition data, and transmit this data to a server in real time.

[1944] Step 6:

[1945] The server analyzes user reaction data and uses generative AI to evolve the artwork.

[1946] Input: User response data obtained in step 5.

[1947] Output: Evolved artwork data.

[1948] How it works: The server analyzes the areas of interest and changes in emotions of the user, and the generative AI modifies the artwork based on that data, for example by emphasizing certain colors or shapes.

[1949] Step 7:

[1950] The device updates and reflects the evolved artwork in the virtual reality space.

[1951] Input: The evolved artwork data generated in step 6.

[1952] Output: An updated artwork in a virtual reality space.

[1953] What it does: The device updates the artwork already displayed in real time, providing the user with a new art experience.

[1954] Step 8:

[1955] The server collects user feedback and ratings and stores them in the generative AI's learning database.

[1956] Input: User feedback and ratings.

[1957] Output: Feedback data stored in the training database.

[1958] Specific operation: After the exhibition ends, the terminal displays a feedback form to the user. The user's ratings and comments are collected, and the server provides this to the generation AI as learning data.

[1959] Step 9:

[1960] The terminal obtains the user's emotional data in real time and further adjusts the artwork based on the user's emotional state.

[1961] Input: User facial and vocal emotion data.

[1962] Output: A coordinated piece of art.

[1963] How it works: The device analyzes the user's emotions based on data obtained from camera footage and microphone input, and then enhances or modifies the artwork based on that.

[1964] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1965] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1966] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1967] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1968] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1969] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1970] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1971] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1972] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1973] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1974] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1975] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1976] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1977] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1978] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1979] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1980] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1981] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1982] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1983] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1984] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1985] The following is further disclosed regarding the above embodiment.

[1986] (Claim 1)

[1987] a means for obtaining user profile data;

[1988] a means for using generative AI to analyze a user's artistic preferences based on the obtained profile data; and

[1989] A means to render artwork generated by generative AI in VR space,

[1990] A means of capturing data in real time based on user gaze tracking and gesture recognition;

[1991] A means of analyzing the acquired data and using generative AI to evolve the artwork;

[1992] A way to update and reflect evolved artworks in VR space,

[1993] A system including:

[1994] (Claim 2)

[1995] a means for obtaining user eye-tracking, gesture recognition, and voice input data;

[1996] A means to store the acquired data and provide it to the generating AI for the next display optimization.

[1997] The system of claim 1 further comprising:

[1998] (Claim 3)

[1999] A means to analyze user response data and evolve the artwork by emphasizing or adding specific art elements;

[2000] A means for displaying a feedback form from users on the evolved artwork;

[2001] A means for storing the feedback data in the learning database of the generating AI;

[2002] The system of claim 1 further comprising:

[2003] "Example 1"

[2004] (Claim 1)

[2005] a means for obtaining user profile data;

[2006] means for using artificial intelligence to analyze user preferences based on the obtained user profile data;

[2007] A means for rendering digital artworks generated by artificial intelligence in a virtual reality space;

[2008] A means of acquiring data in real time based on user gaze tracking and motion recognition;

[2009] A means of using artificial intelligence to analyze the acquired data and evolve the digital artwork;

[2010] A means to update and reflect evolved digital artworks in virtual reality space,

[2011] A system including:

[2012] (Claim 2)

[2013] a means for acquiring user eye tracking, motion recognition, and voice input data;

[2014] A means of storing the acquired data and providing it to the artificial intelligence for the next exhibition optimization;

[2015] The system of claim 1 further comprising:

[2016] (Claim 3)

[2017] A means for analyzing user response data and evolving digital artworks by emphasizing or adding specific art elements;

[2018] A means for displaying a feedback form from users regarding the evolved digital artwork;

[2019] a means for storing the feedback data in an artificial intelligence learning database;

[2020] The system of claim 1 further comprising:

[2021] "Application Example 1"

[2022] (Claim 1)

[2023] a means for obtaining user profile data;

[2024] a means for using generative AI to analyze a user's artistic preferences based on the obtained profile data; and

[2025] A means of rendering artworks generated by generative AI in virtual reality space; and

[2026] A means of capturing data in real time based on user gaze tracking and gesture recognition;

[2027] A means of analyzing the acquired data and using generative AI to evolve the artwork;

[2028] A means to update and reflect evolved artworks in virtual reality space,

[2029] means for providing a terminal for utilizing these means in an autonomous vehicle;

[2030] A system including:

[2031] (Claim 2)

[2032] a means for obtaining user eye-tracking, gesture recognition, and voice input data;

[2033] A means to store the acquired data and provide it to the generating AI for the next display optimization.

[2034] a means for providing access to the vehicle user during the journey within the automated vehicle;

[2035] The system of claim 1 further comprising:

[2036] (Claim 3)

[2037] A means to analyze user response data and evolve the artwork by emphasizing or adding specific art elements;

[2038] A means for displaying a feedback form from users on the evolved artwork;

[2039] A means for storing the feedback data in the learning database of the generating AI;

[2040] a means for soliciting and collecting user feedback within the automated driving vehicle; and

[2041] The system of claim 1 further comprising:

[2042] "Example 2: Combining Emotion Engines"

[2043] (Claim 1)

[2044] A means for acquiring user attribute information;

[2045] a means for using a generative AI to analyze the user's artistic preferences based on the acquired attribute information;

[2046] A means for rendering artwork generated by generative AI in a virtual reality space; and

[2047] A means of acquiring data in real time based on user gaze tracking and motion recognition;

[2048] A means of analyzing the acquired data and using generative AI to evolve the artwork;

[2049] A means to update and reflect evolved artworks in virtual reality space,

[2050] A means for acquiring the user's emotional state in real time and analyzing the data;

[2051] A system including:

[2052] (Claim 2)

[2053] a means for acquiring user eye tracking, motion recognition, and voice input data;

[2054] A means to store the acquired data and provide it to the generating AI for the next display optimization;

[2055] means for analyzing the user's emotion data and further adjusting the artwork based on the user's emotion;

[2056] The system of claim 1 further comprising:

[2057] (Claim 3)

[2058] A means for analyzing user response data and evolving the artwork by emphasizing or adding specific artistic elements;

[2059] a means for displaying a user evaluation form for the evolved artwork;

[2060] A means for storing the evaluation data in the learning database of the generation AI;

[2061] The system of claim 1 further comprising:

[2062] "Application example 2 when combining emotion engines"

[2063] (Claim 1)

[2064] a means for obtaining user profile data;

[2065] a means for using generative AI to analyze a user's artistic preferences based on the obtained profile data; and

[2066] A means of rendering artworks generated by generative AI in virtual reality space; and

[2067] A means of capturing user gaze tracking, gesture recognition, voice input and emotion data in real time;

[2068] A means of using generative AI to analyze the acquired data and evolve the artwork;

[2069] A means to update and reflect evolved artworks in virtual reality space,

[2070] a means of enhancing or adjusting artwork based on user emotional data;

[2071] A means of collecting user feedback and storing it in the generative AI's learning database;

[2072] A system including:

[2073] (Claim 2)

[2074] a means for obtaining user eye-tracking, gesture recognition, and voice input data;

[2075] A means to store the acquired data and provide it to the generating AI for the next display optimization.

[2076] The system of claim 1 further comprising:

[2077] (Claim 3)

[2078] A means to analyze user response data and evolve the artwork by emphasizing or adding specific art elements;

[2079] A means for displaying a feedback form from users on the evolved artwork;

[2080] A means for storing the feedback data in the learning database of the generating AI;

[2081] The system of claim 1 further comprising: [Explanation of symbols]

[2082] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for obtaining user profile data; a means for using generative AI to analyze a user's artistic preferences based on the obtained profile data; and A means to render artwork generated by generative AI in VR space, A means of capturing data in real time based on user gaze tracking and gesture recognition; A means of analyzing the acquired data and using generative AI to evolve the artwork; A way to update and reflect evolved artworks in VR space, A system including:

2. a means for obtaining user eye-tracking, gesture recognition, and voice input data; A means to store the acquired data and provide it to the generating AI for the next display optimization. The system of claim 1 further comprising:

3. A means to analyze user response data and evolve the artwork by emphasizing or adding specific art elements; A means for displaying a feedback form from users on the evolved artwork; A means for storing the feedback data in the learning database of the generating AI; The system of claim 1 further comprising:

Images