System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving, by a server, a photograph or a moving image acquired from a user device and analyzing a scene, a person, and a motion from the photograph or the moving image; means for generating, by a generation AI, a three dimensional Metaverse space based on a result of the analysis; and means for storing the generated three dimensional Metaverse space and providing the three dimensional Metaverse space so that a user can access the three dimensional Metaverse space.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, users store many memories in the form of photos and videos. However, these memories are merely two-dimensional information, and it is sometimes difficult to fully recreate the emotion and atmosphere of the moment. Furthermore, information that can only be stored in photos and videos often fades or fades from memory over time. To address this issue, a more immersive and realistic way to preserve and experience memories is needed. [Means for solving the problem]

[0005] The present invention provides a system that generates a metaverse space based on photos and videos taken by users on their devices, allowing them to experience their memories in 3D. Specifically, photos and videos are first acquired from the user's device. Next, the server analyzes the received data and extracts scenery, people, and movements. The generation AI then generates a 3D metaverse space based on the analysis results and saves the generated space. Users can access this 3D space using a VR headset or AR-compatible device, allowing for an immersive experience.

[0006] A "user terminal" is a device that a user uses to take photos and videos and send the data to a server, and includes smartphones, tablets, digital cameras, etc.

[0007] A "server" is a computer system that receives, analyzes, generates, and stores photo and video data sent from user terminals.

[0008] "Photos or videos" are still images or video data taken by a user, and are digital media stored in two-dimensional format.

[0009] "Scenery, people, and movement" refer to the visual elements contained in a photograph or video, with scenery referring to nature and buildings, people referring to the human figure, and movement referring to the changes and movements of these elements.

[0010] "Generative AI" is an artificial intelligence algorithm that generates a three-dimensional metaverse space based on features extracted from a photo or video.

[0011] A "three-dimensional metaverse space" is a three-dimensional virtual space that reproduces the characteristics of the real world, and is a digital environment that users can immerse themselves in.

[0012] "Storage" is the process of storing the generated three-dimensional metaverse space in a database so that it can be accessed later.

[0013] A "VR headset" is a device that allows users to immerse themselves in a three-dimensional metaverse space and provides virtual reality primarily through sight and hearing.

[0014] An "AR-enabled device" is a device that can overlay virtual content onto the real world, providing augmented reality.

[0015] An "experience" is the act of a user exploring the three-dimensional metaverse space through sight and sound, and enjoying scenes and events that are reproduced with a sense of realism.

[0016] "Analysis" is the process of extracting necessary features from photos or videos, and uses computer vision and machine learning technologies.

[0017] "Analysis results" are features extracted from a photo or video and information about those features.

[0018] An "HTTP POST request" is a type of Internet communication protocol for sending data from a user terminal to a server.

[0019] "Streaming" is a method of transmitting data from a server to a user terminal in real time, making the data immediately available to the user. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] This invention is a system that recreates the scenery, people, and movements of a moment in a metaverse space based on photos and videos taken by the user. This system utilizes the user's device, a server, and generation AI to provide the user with an immersive experience.

[0042] System configuration

[0043] 1. User Device:

[0044] They act as devices that allow users to take photos and videos, including smartphones, tablets, and digital cameras.

[0045] A dedicated application is installed to allow users to send captured image data to a server.

[0046] Users access the saved metaverse space and use VR headsets or AR-enabled devices to experience it.

[0047] 2. Server:

[0048] It has a component that receives photo and video data sent from the user's device.

[0049] Computer vision techniques are used to analyze the received data and extract scenery, people, and movement.

[0050] Based on the extracted information, a 3D metaverse space is generated using generative AI.

[0051] The generated metaverse spatial data is stored in a database and provided for users to access.

[0052] 3. Generation AI:

[0053] Based on the analysis results, it includes deep learning models and 3D reconstruction algorithms to generate 3D models that reproduce real-world scenes, people, and movements.

[0054] System Operation

[0055] Data acquisition and transmission (user's device)

[0056] User takes a photo or video:

[0057] A user takes photos and videos using the device's camera app while traveling or at an event.

[0058] The captured data is stored in the device's local storage.

[0059] User uploads data:

[0060] The user opens the dedicated app and goes to the upload screen of the metaverse generation service.

[0061] Select the saved photo or video file and press the upload button.

[0062] The app sends the selected file to the server as an HTTP POST request.

[0063] Data reception and analysis (server)

[0064] Server receives data:

[0065] The server receives the HTTP request and retrieves the transmitted file data.

[0066] The received file is stored in a directory for temporary storage.

[0067] The server analyzes the photos and videos:

[0068] The server's image analysis module analyzes the transmitted photo and video data.

[0069] Image analysis uses computer vision libraries such as OpenCV and TensorFlow to recognize scenery and people.

[0070] In video analysis, a video is divided into frames, and each frame is subjected to image analysis to extract movement.

[0071] Generation and storage of 3D metaverse space (server)

[0072] The server generates a 3D metaverse space using generative AI:

[0073] The extracted feature information is input into a generation AI to generate a 3D metaverse space.

[0074] Generative AI uses GANs and 3D reconstruction algorithms to recreate realistic situations.

[0075] The server saves the generated metaverse space:

[0076] The generated 3D data is stored in a database along with metadata.

[0077] Generate an identifier (ID) for the stored spatial data and link it to the user's profile.

[0078] Providing a metaverse experience (user device)

[0079] User accesses the metaverse space:

[0080] The user puts on a VR headset or AR-compatible device and launches a dedicated app.

[0081] Log in to the server from the app and view a list of saved metaverse spaces.

[0082] Select a specific metaverse space from the list provided to begin the experience.

[0083] The app streams selected metaverse spatial data from the server and displays it to the user in real time.

[0084] Specific examples

[0085] 1. Relive your travel memories:

[0086] A user takes a photo or video in front of the Eiffel Tower while traveling in Paris (user's device).

[0087] The taken photo is uploaded by dragging and dropping it onto the upload screen of the metaverse generation service (user's device).

[0088] The server receives the photo data sent from the user's terminal and temporarily stores it (server).

[0089] The server's image analysis module recognizes the Eiffel Tower and the surrounding scenery from the sent photo and extracts its features (server).

[0090] The generative AI generates a 3D model of the Eiffel Tower based on the extracted features and recreates the background scenery (server).

[0091] The generated 3D model and scenery data are stored in a database and linked to the user's profile (server).

[0092] Users put on a VR headset at home and open a dedicated app to access the Eiffel Tower metaverse space (on their device).

[0093] Users can move freely within the metaverse and experience the Eiffel Tower and its surroundings in 360 degrees (on their device).

[0094] 2. Recreating child growth records:

[0095] A user records a video of their child's first birthday (on their device).

[0096] The captured video is uploaded to the metaverse generation service (user's device).

[0097] The server receives the video and analyzes the birthday cake and the child's movements (server).

[0098] Based on the analyzed information, the generation AI generates a 3D metaverse space for the birthday party (server).

[0099] The generated 3D data is stored in a database and made accessible to users (server).

[0100] The user uses a VR headset to relive their child's first birthday (on the user's device).

[0101] The system of the present invention allows users to realistically preserve and experience memories in a highly immersive alternative to two-dimensional photos and videos.

[0102] The processing flow will be explained below.

[0103] Step 1:

[0104] Data acquisition (user device)

[0105] Users use the camera app on their devices to take photos or videos of scenery at their travel destinations or events.

[0106] The captured data is stored in the device's local storage.

[0107] Step 2:

[0108] Data transmission (user's device)

[0109] Users open the dedicated app and select the photos and videos they have taken.

[0110] When you press the upload button, the app sends the selected data to the server as an HTTP POST request.

[0111] Step 3:

[0112] Data reception (server)

[0113] The server receives an HTTP POST request to retrieve the photo or video file.

[0114] To temporarily store the received data, save it in the specified directory.

[0115] Step 4:

[0116] Image and video analysis (server)

[0117] The server uses computer vision libraries (e.g., OpenCV and TensorFlow) to analyze the stored photo and video data.

[0118] In the case of images, the server extracts features such as scenery, people, and objects from the image.

[0119] In the case of video, the server divides the video into frames, analyzes each frame, and extracts movement.

[0120] Step 5:

[0121] 3D Metaverse Space Generation (Server)

[0122] Based on the analysis results, the server generates a 3D metaverse space using generative AI (e.g., GAN or 3D reconstruction algorithms).

[0123] Generative AI uses extracted features to recreate scenes, people, and movements.

[0124] Step 6:

[0125] Metaverse space storage (server)

[0126] The server stores the generated 3D metaverse spatial data in a database.

[0127] Generate an identifier (ID) for the saved space and link it to the user's profile.

[0128] Step 7:

[0129] Providing a metaverse experience (user device)

[0130] The user puts on a VR headset or AR-compatible device and launches a dedicated app.

[0131] Users log in to the server through the app and view a list of saved metaverse spaces.

[0132] Select a specific metaverse space from the list provided to begin the experience.

[0133] The app streams selected metaverse spatial data in real time from the server and displays it to the user.

[0134] In this way, users can experience an immersive three-dimensional metaverse space generated from two-dimensional photos and videos.

[0135] Example 1

[0136] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0137] Conventional recording methods using photographs and videos forced users to rely on two-dimensional information, making it difficult to recreate the atmosphere or three-dimensional experience of the scene. Furthermore, there were limited ways to preserve detailed memories of specific events or travel destinations and experience them later in real life. Furthermore, existing systems required complex processes for data analysis and reproduction, making practical application difficult.

[0138] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0139] In this invention, the server includes: means for receiving photos or videos acquired from a user terminal and analyzing scenery, people, and movements from the photos or videos; means for a generation AI to generate a 3D metaverse space based on the analysis results; means for saving the generated 3D metaverse space and providing it for user access; means for a user to take photos or videos and upload the data to the server using a dedicated application; means for the server to save the transmitted photos or videos in a temporary directory and analyze them using computer vision technology; and means for saving the generated metaverse space data in a database and linking the identifier to a user profile. This allows users to have a more immersive experience, enabling the realistic reproduction of on-site atmospheres and three-dimensional experiences.

[0140] "User terminal" refers to any digital device that allows a user to take and save photos and videos, and specific examples include smartphones and digital cameras.

[0141] A "server" refers to a computer system that receives, analyzes, and stores data sent from a user terminal, and provides various services to the user.

[0142] "Photos or videos" refers to still images or video data taken by a user using a digital device.

[0143] "Analysis" refers to the process of recognizing scenery, people, and movements from photos or videos and extracting their respective characteristics.

[0144] "Scenery" refers to the natural or man-made scenery or backgrounds contained in photographs or videos.

[0145] "People" refers to human figures appearing in photographs and videos.

[0146] "Motion" refers to the movement and activity of people and objects in a photo or video.

[0147] "Generative AI" refers to artificial intelligence techniques used to generate 3D Metaverse spaces based on analyzed data, including GANs and 3D reconstruction algorithms.

[0148] "3D Metaverse space" refers to a digital space in which real-world scenery, people, and movements are reproduced in three dimensions within a virtual space.

[0149] "Dedicated application" refers to software that users use to upload photos and videos to a server.

[0150] The "temporary directory" refers to a folder where the server temporarily stores data received.

[0151] "Computer vision technology" refers to the technology that allows computers to analyze images and videos and extract information. Examples include OpenCV and TensorFlow.

[0152] "Database" refers to a system for efficiently storing and managing the generated metaverse spatial data. A specific example is MongoDB.

[0153] "Identifier" refers to an ID that uniquely identifies stored metaverse spatial data.

[0154] "User profile" refers to a digital data structure for storing information and data related to a user.

[0155] "VR headset" refers to a device used by a user to experience a virtual reality space, including, for example, Oculus Rift.

[0156] "AR-enabled device" means a device used to experience an augmented reality space, including, for example, Microsoft HoloLens.

[0157] MODE FOR CARRYING OUT THE INVENTION

[0158] This invention is a system that recreates the scenery, people, and movements of a moment in a metaverse space based on photos and videos taken by the user. This system utilizes the user's device, a server, and generation AI to provide the user with an immersive experience.

[0159] 1. System Configuration

[0160] User terminal

[0161] A user device is a general term for a digital device that allows a user to take and save photos and videos, including smartphones and digital cameras. A dedicated application (e.g., the Metaverse Creator app) is installed on the device to transmit the captured data to a server. Users can access and experience the saved Metaverse space using a VR headset (e.g., Oculus Rift) or an AR-enabled device (e.g., Microsoft HoloLens).

[0162] server

[0163] The server receives photo and video data sent from user devices, analyzes it, and generates the metaverse space. The server includes the following main components:

[0164] Data receiving module: Receives HTTP requests and retrieves the transmitted file data.

[0165] Temporary storage: The directory where received data is temporarily stored (e.g. / tmp / uploads).

[0166] Image analysis module: Uses computer vision libraries such as OpenCV and TensorFlow to extract scenery, people, and movement from photo and video data.

[0167] Generative AI module: Uses GAN and 3D reconstruction algorithms to generate a 3D metaverse space based on extracted data.

[0168] Database: A database (e.g., MongoDB) to store the generated metaverse spatial data and associated metadata.

[0169] 2. System Operation

[0170] Data acquisition and transmission by the user's device

[0171] A user takes photos and videos using a smartphone or digital camera. For example, while traveling, the user takes a photo of the Eiffel Tower using the iPhone camera app and saves it to the camera roll. The user then launches the MetaverseCreator app, selects the photo, and uploads it. The app then sends the photo data to the server as an HTTP POST request.

[0172] Data reception and analysis by the server

[0173] The server receives the HTTP request and saves the data in a temporary directory. The image analysis module then analyzes the photo data, recognizing scenery such as the Eiffel Tower, people, and movements, and extracting feature points. This feature point information is stored as JSON data.

[0174] Creation and storage of 3D metaverse space

[0175] The server inputs the extracted feature information into a generative AI module, which then uses a GAN model and 3D reconstruction algorithm to generate a 3D structure of the Eiffel Tower and recreate the background scenery. The generated metaverse spatial data is stored in a database, and an identifier is generated and linked to the user's profile.

[0176] User experience of the metaverse space

[0177] Users put on a VR headset or AR-enabled device and launch the MetaverseViewer app. They log in within the app and view a list of Metaverse spaces linked to their profile. They select a specific Metaverse space and begin the experience. The app streams the Metaverse space data from the server and displays it on the user's device in real time.

[0178] Examples and prompts

[0179] For example, if a user uploads a photo of the Eiffel Tower taken during a trip to Paris, and a generative AI generates a 3D metaverse of the Eiffel Tower based on that photo, the actual prompt would look like this:

[0180] "Generate a 3D model of the Eiffel Tower based on the following photo."

[0181] "Reconstruct a scene from a child's birthday party using frames from this video."

[0182] The system of the present invention allows users to realistically preserve and experience memories in a highly immersive alternative to two-dimensional photos and videos.

[0183] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0184] Step 1:

[0185] A user takes a photo or video. A user takes a photo or video using a smartphone or digital camera. For example, while traveling, a user takes a photo of the Eiffel Tower using the camera app on their iPhone and saves it to their camera roll. The input is the camera app on the digital device, and the output is the photo or video data saved in local storage.

[0186] Step 2:

[0187] The user uploads data to the server using a dedicated application. The user launches the MetaverseCreator app, selects the photos and video files they have taken, and uploads them. The input is the photo or video file saved in local storage, and the output is the file data sent as an HTTP POST request. During this process, an upload progress bar is displayed.

[0188] Step 3:

[0189] The server receives the data. The server receives the HTTP request and saves the submitted file data and metadata in a temporary directory (e.g., / tmp / uploads). The input is the photo or video data included in the HTTP request body, and the output is the file saved in the temporary directory.

[0190] Step 4:

[0191] The server analyzes photos and videos. The server's image analysis module (e.g., OpenCV, TensorFlow) analyzes the photo and video data stored in a temporary directory. The input is the file stored in the temporary directory, and the output is feature information about the scenery, people, and actions stored in JSON format. Specifically, feature points of the Eiffel Tower and people are extracted from photos, and each frame of the video is analyzed to detect movement.

[0192] Step 5:

[0193] The server uses generative AI to generate a 3D metaverse space. The server inputs the extracted feature point information into a generative AI module (e.g., GAN or 3D reconstruction algorithm) to generate the 3D metaverse space. The input is JSON data of the feature point information, and the output is data in the generated 3D metaverse space. For example, a GAN model generates the 3D structure of the Eiffel Tower and recreates the background scenery.

[0194] Step 6:

[0195] The server saves the generated metaverse space in a database. The server saves the generated metaverse space data in a database (e.g., MongoDB) and links the identifier to the user profile. The input is the 3D metaverse space data and the user ID, and the output is the data saved in the database and the generated identifier.

[0196] Step 7:

[0197] A user accesses the Metaverse space by putting on a VR headset (e.g., Oculus Rift) or an AR-enabled device (e.g., Microsoft HoloLens) and launching the MetaverseViewer app. The input is the application on the device, and the output is the Metaverse space data streamed from the server. When the user selects a Metaverse space in the app and presses the "Start Experience" button, the app displays the data in real time, and the user enjoys an immersive experience.

[0198] (Application example 1)

[0199] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0200] There is a need for a method that allows users to upload photos and videos they have taken to the metaverse space through a dedicated application, and then share the 3D metaverse space generated from it with other users, thereby providing a visually immersive experience. However, existing technologies do not easily enable the generation and sharing of such advanced metaverse spaces, so a new solution is needed.

[0201] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0202] In this invention, the server includes means for receiving photos or videos acquired from a user terminal and analyzing scenery, people, and movements from the photos or videos, means for a generation AI to generate a 3D metaverse space based on the analysis results, means for saving the generated 3D metaverse space and providing it for users to access, means for users to upload and share photos or videos using a smartphone application, and means for sharing the generated 3D metaverse space with friends and family. This allows users to generate realistic 3D metaverse spaces based on photos and videos they have taken and easily share them with other users, providing a visually realistic and immersive experience.

[0203] "User terminal" refers to a device that takes photos and videos and has installed an application for sending that data to a server.

[0204] A "server" refers to a computer system that receives data sent from a user terminal, analyzes it, and performs the necessary processing.

[0205] "Generative AI" refers to artificial intelligence technology for generating a three-dimensional metaverse space based on the analysis of photos and videos sent from user devices.

[0206] "3D Metaverse space" refers to a three-dimensional virtual space generated based on real-world scenery, people, and movements.

[0207] "Smartphone application" refers to software that is installed on a smartphone and allows users to upload photos and videos, share and experience the metaverse space.

[0208] "Analyzing" refers to the process of extracting specific information (scenery, people, movements, etc.) from photos and videos.

[0209] "Saving" refers to the act of storing the generated 3D metaverse spatial data in a state that allows it to be accessed later.

[0210] "Providing" refers to the process of allowing a user to access a saved three-dimensional metaverse space.

[0211] "Sharing" refers to the act of exchanging the generated 3D metaverse space with other users in an easily accessible manner.

[0212] "Friends and Family" refers to close people with whom the user wants to share.

[0213] This invention is a system that recreates the scenery, people, and movements of a moment in a 3D metaverse space based on photos and videos taken by the user. This system utilizes the user's device, a server, and a generation AI to provide the user with an immersive experience.

[0214] System configuration

[0215] It consists of the following main components:

[0216] 1. User Device:

[0217] Camera device: A device that allows a user to take photos or videos. Examples include smartphones, tablets, and digital cameras.

[0218] Dedicated application: An application is installed to send the captured data to the server. VR headsets and AR-compatible devices are also used to access and experience the generated 3D metaverse space.

[0219] 2. Server:

[0220] Receiving module: Contains a component that receives and temporarily stores photo and video data sent from the user device.

[0221] Image analysis module: Analyzes received photo and video data using computer vision technology, specifically using libraries such as OpenCV and TensorFlow.

[0222] Generative AI module: Generates a 3D metaverse space based on the extracted information. This uses deep learning models, including GANs and 3D reconstruction algorithms.

[0223] Database: Stores the generated 3D metaverse spatial data and makes it accessible to users.

[0224] 3. Generation AI:

[0225] Deep learning model: A technology for generating 3D models that reproduce real-world scenery, people, and movements based on analysis results sent from the user's device.

[0226] 3D reconstruction algorithm: Creates a realistic 3D space based on the analysis results.

[0227] System Operation

[0228] 1. Data acquisition and transmission (user terminal):

[0229] Users take photos and videos using their smartphones and upload the data to the server via a dedicated application. Uploads are made via HTTP POST requests.

[0230] 2. Data reception and analysis (server):

[0231] The server receives and temporarily stores the photos and video data sent from the user's device. It then uses computer vision technology to analyze the scenery, people, and movements from the received data. Libraries such as OpenCV and TensorFlow are used for this.

[0232] 3. Generation and storage of 3D metaverse space (server):

[0233] The analysis results are input into the generation AI to generate a 3D metaverse space. The generated 3D data is stored in a database and made available for users to access.

[0234] 4. Providing a Metaverse experience (user device):

[0235] Users can access the generated 3D metaverse space through a dedicated application using a VR headset or AR-compatible device, and can also share the generated metaverse space with friends and family.

[0236] Specific examples

[0237] Travel Reenactment:

[0238] Users upload photos taken in front of the Eiffel Tower during their travels. The server analyzes the photos and uses generative AI to generate a 3D metaverse of the Eiffel Tower and its surroundings. Users can access this metaverse by wearing a VR headset and enjoy a 360-degree experience.

[0239] Recreating a child's growth record:

[0240] Users upload videos of their child's first birthday. The server analyzes the videos and uses generative AI to generate a 3D space of the birthday party. Users can access this recreated space and relive their child's growth and memories.

[0241] Prompt Sentence Examples

[0242] "Upload a photo of the Eiffel Tower and generate a realistic 3D metaverse space based on this photo."

[0243] This system allows users to easily recreate their own photos and videos in a 3D metaverse space and share that space with others, providing a visually immersive experience.

[0244] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0245] Step 1:

[0246] The user takes a photo or video

[0247] Users take photos and videos using the camera app on their smartphones.

[0248] Input: Photo or video data from a smartphone camera app

[0249] Output: Photo or video files stored in the smartphone's local storage

[0250] Step 2:

[0251] User uploads data

[0252] The user opens the application, selects the photos and video files they have taken, and presses the upload button, which sends these files to the server as an HTTP POST request.

[0253] Input: Photo or video files selected through a dedicated application, server upload endpoint

[0254] Output: HTTP POST request sent to the server

[0255] Step 3:

[0256] The server receives the data

[0257] The server receives the HTTP POST request, retrieves the sent file data, and saves it in a temporary directory.

[0258] Input: Photo or video data received via an HTTP POST request

[0259] Output: File data saved in temporary directory

[0260] Step 4:

[0261] The server analyzes the photos and videos.

[0262] The server's image analysis module uses OpenCV and TensorFlow to analyze the received photo and video data. Image analysis recognizes scenery and people, while video analysis divides the video into frames to extract movement.

[0263] Input: Photo or video data saved in the temporary directory

[0264] Output: Scenery, people, and movement information as analysis results

[0265] Step 5:

[0266] The server generates a 3D metaverse space using generation AI.

[0267] The server inputs the analysis results into the generation AI, which then generates a 3D metaverse space using GAN and 3D reconstruction algorithms.

[0268] Input: Scenery, people, and movement information as analysis results

[0269] Output: Generated 3D metaverse spatial data

[0270] Step 6:

[0271] The server stores the generated 3D metaverse spatial data.

[0272] The server stores the generated 3D metaverse spatial data in a database and makes it accessible to users.

[0273] Input: Generated 3D metaverse spatial data, user identification information

[0274] Output: 3D metaverse spatial data stored in a database, metaverse space linked to user profile

[0275] Step 7:

[0276] A user accesses the metaverse space

[0277] Users use a VR headset or AR-compatible device to log in through a dedicated application, select one from a list of saved 3D metaverse spaces, and begin the experience.

[0278] Input: User login information, list of 3D metaverse spaces stored in the database

[0279] Output: 3D metaverse space streamed to user's device

[0280] Step 8:

[0281] Users share the metaverse space

[0282] Users can share the 3D metaverse space they create with friends and family through a dedicated application, generating a shared link and granting access rights to specific users.

[0283] Input: 3D metaverse space linked to user profile, shared identity information

[0284] Output: Shared link and access rights to the shared 3D metaverse space

[0285] The above are the specific processing steps in the system for implementing this invention. At each step, various data processing and calculations are performed based on the input data, and the results are passed on to the next step, thereby operating the entire system.

[0286] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0287] The present invention is a system that recreates the scenery, people, and movements of a user in a metaverse space based on photos and videos taken by the user, and further adjusts the experience by recognizing the user's emotions. The following describes the detailed operation of the system.

[0288] System configuration

[0289] 1. User Device:

[0290] They act as devices that allow users to take photos and videos, including smartphones, tablets, and digital cameras.

[0291] A dedicated application is installed to allow users to send captured image data to a server.

[0292] It also has the function of capturing the user's facial expressions and voice and sending them to the emotion engine.

[0293] Users access the saved metaverse space and use VR headsets or AR-enabled devices to experience it.

[0294] 2. Server:

[0295] It has a component that receives photo and video data sent from the user's device, as well as emotion data recognized by the emotion engine.

[0296] Computer vision techniques are used to analyze the received data and extract scenery, people, and movement.

[0297] Based on the extracted information, a 3D metaverse space is generated using generative AI.

[0298] The generated metaverse spatial data is stored in a database and provided for users to access.

[0299] Tailor your metaverse experience in real time based on data from the emotion engine.

[0300] 3. Generation AI:

[0301] Based on the analysis results, it includes deep learning models and 3D reconstruction algorithms to generate 3D models that reproduce real-world scenes, people, and movements.

[0302] 4. Emotion Engine:

[0303] Recognizes emotions by analyzing the user's facial expressions and voice.

[0304] Adjust visual and audio effects within the metaverse space based on the emotions recognized.

[0305] System Operation

[0306] Data acquisition and transmission (user's device)

[0307] User takes a photo or video:

[0308] A user takes photos and videos using the device's camera app while traveling or at an event.

[0309] The captured data is stored in the device's local storage.

[0310] User uploads data:

[0311] The user opens the dedicated app and selects the photos and videos they have taken.

[0312] When you press the upload button, the app sends the selected data to the server as an HTTP POST request.

[0313] Data reception and analysis (server)

[0314] Server receives data:

[0315] The server receives an HTTP POST request to retrieve the photo or video file.

[0316] To temporarily store the received data, save it in the specified directory.

[0317] The server analyzes the photos and videos:

[0318] The server's image analysis module analyzes the transmitted photo and video data.

[0319] Image analysis uses computer vision libraries such as OpenCV and TensorFlow to recognize scenery and people.

[0320] In video analysis, a video is divided into frames, and each frame is subjected to image analysis to extract movement.

[0321] 3D Metaverse Space Generation (Server)

[0322] The server generates a 3D metaverse space using generative AI:

[0323] The extracted feature information is input into a generation AI to generate a 3D metaverse space.

[0324] Generative AI uses GANs and 3D reconstruction algorithms to recreate realistic situations.

[0325] Metaverse space storage (server)

[0326] The server stores the generated 3D metaverse space:

[0327] The server stores the generated 3D metaverse spatial data in a database.

[0328] Generate an identifier (ID) for the stored spatial data and link it to the user's profile.

[0329] Emotion recognition and user experience adjustment (emotion engine and server)

[0330] Emotion engine recognizes emotions:

[0331] The user device captures the user's facial expressions and voice and sends them to the emotion engine.

[0332] The emotion engine analyzes this data and recognizes the user's emotional state.

[0333] Emotion-based experience tailoring:

[0334] The emotion data obtained from the emotion engine is received by the server.

[0335] The server changes the visual and sound effects in the metaverse space in real time based on the emotional data.

[0336] For example, if the user is impressed, the scenery will be made more vivid and the background music will be changed to something more exciting.

[0337] Providing a metaverse experience (user device)

[0338] User accesses the metaverse space:

[0339] The user puts on a VR headset or AR-compatible device and launches a dedicated app.

[0340] Users log in to the server through the app and view a list of saved metaverse spaces.

[0341] Select a specific metaverse space from the list provided to begin the experience.

[0342] The app streams selected metaverse spatial data in real time from the server and displays it to the user.

[0343] Specific examples

[0344] 1. Travel memory recall and emotional regulation:

[0345] A user takes a photo or video in front of the Eiffel Tower while traveling in Paris (user's device).

[0346] The taken photo is uploaded by dragging and dropping it onto the upload screen of the metaverse generation service (user's device).

[0347] The server receives the photo data sent from the user's device and temporarily stores it (server).

[0348] The server's image analysis module recognizes the Eiffel Tower and the surrounding scenery from the sent photo and extracts its features (server).

[0349] The generative AI generates a 3D model of the Eiffel Tower based on the extracted features and recreates the background scenery (server).

[0350] The generated 3D model and scenery data are stored in a database and linked to the user's profile (server).

[0351] Users put on a VR headset at home and open a dedicated app to access the Eiffel Tower metaverse space (on their device).

[0352] While the user is experiencing the experience, the emotion engine analyzes the user's facial expressions and if it recognizes emotion, the server adjusts the effects and changes the background music to an emotional one (server).

[0353] 2. Recapitulation of child development records and emotional regulation:

[0354] A user records a video of their child's first birthday (on their device).

[0355] The captured video is uploaded to the metaverse generation service (user's device).

[0356] The server receives the video and analyzes the birthday cake and the child's movements (server).

[0357] Based on the analyzed information, the generation AI generates a 3D metaverse space for the birthday party (server).

[0358] The generated 3D data is stored in a database and made accessible to users (server).

[0359] The user uses a VR headset to relive their child's first birthday (on the user's device).

[0360] If the emotion engine recognizes the user's joy, the server will brighten the lights in the space and make the background music more enjoyable (server).

[0361] The system of the present invention allows users to realistically preserve memories in a highly immersive alternative to two-dimensional photos and videos, and allows for customized experiences that reflect their emotional state at any given time.

[0362] The processing flow will be explained below.

[0363] Step 1:

[0364] Data acquisition (user device)

[0365] Users use the camera app on their devices to take photos or videos of scenery at their travel destinations or events.

[0366] The captured data is stored in the device's local storage.

[0367] Step 2:

[0368] Data transmission (user's device)

[0369] Users open the dedicated app and select the photos and videos they have taken.

[0370] When you press the upload button, the app sends the selected data to the server as an HTTP POST request.

[0371] Step 3:

[0372] Data reception (server)

[0373] The server receives an HTTP POST request to retrieve the photo or video file.

[0374] To temporarily store the received data, save it in the specified directory.

[0375] Step 4:

[0376] Image and video analysis (server)

[0377] The server uses computer vision libraries (e.g., OpenCV and TensorFlow) to analyze the stored photo and video data.

[0378] In the case of images, the server extracts features such as scenery, people, and objects from the image.

[0379] In the case of video, the server divides the video into frames, analyzes each frame, and extracts movement.

[0380] Step 5:

[0381] 3D Metaverse Space Generation (Server)

[0382] Based on the analysis results, the server generates a 3D metaverse space using generative AI (e.g., GAN or 3D reconstruction algorithms).

[0383] Generative AI uses extracted features to recreate scenes, people, and movements.

[0384] Step 6:

[0385] Metaverse space storage (server)

[0386] The server stores the generated 3D metaverse spatial data in a database.

[0387] Generate an identifier (ID) for the saved space and link it to the user's profile.

[0388] Step 7:

[0389] Data acquisition for emotion recognition (user device)

[0390] The user puts on a VR headset or AR-compatible device and launches the dedicated app.

[0391] Sensors and microphones begin to operate to capture the user's facial expressions and voice.

[0392] Step 8:

[0393] Sending emotion data (user device)

[0394] The user's facial expression and voice data are sent to the server in real time.

[0395] This data is then forwarded to a dedicated endpoint to be analyzed by the emotion engine.

[0396] Step 9:

[0397] Receiving and analyzing emotion data (server and emotion engine)

[0398] The server passes the received facial expression and voice data to the emotion engine.

[0399] The emotion engine analyzes this data and recognizes the user's emotional state.

[0400] Step 10:

[0401] Emotion-based experience adjustment (server)

[0402] The server adjusts the visual and sound effects in the metaverse space in real time based on the emotional state data obtained from the emotion engine.

[0403] For example, if the user is moved, the scenery is made more vivid and the background music is changed to something more moving.

[0404] Step 11:

[0405] Providing a metaverse experience (user device)

[0406] Users log in to the server through a dedicated app and view a list of saved metaverse spaces.

[0407] Select a specific metaverse space from the list provided to begin the experience.

[0408] The app streams selected metaverse spatial data in real time from the server and displays it to the user.

[0409] In this way, users can experience an immersive three-dimensional metaverse space generated from two-dimensional photos and videos, and further customize it to suit their emotions at the time.

[0410] Example 2

[0411] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0412] Conventional methods simply record photos and videos, but have the problem of not being able to reflect or reproduce the user's emotions and experiences in real time. Furthermore, simple 2D data lacks a sense of immersion, making it difficult to reproduce the emotions and joy that users actually felt at the time.

[0413] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0414] In this invention, the server includes means for receiving photos or videos acquired from a user terminal and analyzing scenery, people, and movements from the photos or videos, means for a generation AI to generate a 3D metaverse space based on the analysis results, means for saving the generated 3D metaverse space and providing it for user access, means for capturing the user's facial expressions and voice and transmitting them to an emotion engine that recognizes emotions, and means for adjusting visual and audio effects in the metaverse space in real time based on the recognized emotion data. This allows users to realistically recreate past experiences with a high level of immersion and have a customized experience that matches their emotions at the time.

[0415] "User terminal" refers to a device that a user uses to take photos and videos and upload the data to a server, and includes smartphones, tablets, digital cameras, etc.

[0416] "Server" refers to a computer system that receives data sent from a user terminal, analyzes, processes, stores, and provides the data.

[0417] "Photos or videos" refers to still images or video files taken by a user using a device.

[0418] "Analysis of scenery, people, and movements" refers to the process of using computer vision technology to recognize scenery, people, and movements in photographs or video data and extract their characteristics.

[0419] "Generative AI" refers to deep learning models and 3D reconstruction algorithms for generating a 3D metaverse space based on extracted feature information.

[0420] "3D Metaverse space" refers to a three-dimensional virtual space generated by a generation AI to recreate real-world situations as virtual environments.

[0421] An "emotion engine" refers to software or hardware that analyzes a user's facial expressions and voice and recognizes the user's emotional state.

[0422] "Real-time adjustment of visual and audio effects" refers to the process of dynamically changing the images and audio displayed within the metaverse space to match the user's emotions based on recognized user emotional data.

[0423] "VR headset" refers to a device worn on the head to immerse a user in a virtual reality environment.

[0424] "AR-compatible device" refers to a device that uses augmented reality technology to overlay virtual objects on the real world.

[0425] "Computer vision technology" refers to technology that enables computers to analyze and understand the content of images and videos, and includes libraries such as OpenCV and TensorFlow.

[0426] This invention is a system that recreates the scenery, people, and movements of a moment in a 3D metaverse space based on photos and videos taken on a user's device, and adjusts the experience by recognizing the user's emotions. The detailed operation of the system is explained below.

[0427] A user takes a photo or video using a user device such as a smartphone, tablet, or digital camera. This data is temporarily stored on the user's device. The user then uses a dedicated application to select the photos or videos they have taken and upload them to the server. Uploads are made as HTTP POST requests.

[0428] The server temporarily stores the received photo or video data in a specified directory. The server's image analysis module then analyzes the data to recognize and extract scenery, people, and movements. This analysis uses computer vision technologies such as OpenCV and TensorFlow.

[0429] Based on the analysis results, the server uses generative AI to generate a 3D metaverse space. This generative AI uses GAN (generative adversarial networks) and 3D reconstruction algorithms to create a realistic virtual space. The generated 3D metaverse space data is then stored in a database, and an identifier is generated and linked to the user's profile.

[0430] Users put on a VR headset or AR-compatible device and log in to the server through a dedicated application. A list of saved 3D metaverse spaces is displayed, and the user selects a specific space to begin the experience. The selected space data is streamed in real time by the server and displayed on the user's device.

[0431] During the experience, the user's device captures facial expressions and voice and sends this data to the emotion engine. The emotion engine analyzes the user's emotional state and sends the recognized emotional data to the server. The server uses this data to adjust the visual and audio effects in the metaverse space in real time. For example, if the user is emotional, the scenery will be made more vivid and the background music will be more exciting.

[0432] For example, a user takes a photo in front of the Eiffel Tower while traveling in Paris and uploads the photo data to a server using a dedicated app. Based on this, the server generates a 3D metaverse space, which the user can access. The emotion engine recognizes the user's emotions during the experience, and the server adjusts the background music accordingly.

[0433] An example of a prompt sentence is as follows:

[0434] "Generate and display a 3D metaverse of the Eiffel Tower based on photos from a trip to Paris."

[0435] In this way, the system of the present invention allows users to realistically preserve memories in a highly immersive alternative to two-dimensional photos and videos, and allows for customized experiences based on emotional state.

[0436] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0437] Step 1:

[0438] The user takes a photo or video.

[0439] Action: The user takes a photo using a smartphone or digital camera.

[0440] Input: Real-world scenes and people.

[0441] Output: Captured photo or video files (formats such as JPEG or MP4).

[0442] Step 2:

[0443] The user uploads the captured data to the server.

[0444] How it works: Use the dedicated app to select the shooting data and press the upload button.

[0445] Input: The captured photo or video file.

[0446] Output: Sent as an HTTP POST request to the server.

[0447] Step 3:

[0448] The server receives and stores the data.

[0449] How it works: The server receives an HTTP POST request and temporarily saves the photos and videos to the specified directory.

[0450] Input: The photo or video file included in the HTTP POST request.

[0451] Output: Temporarily saved photo or video files.

[0452] Step 4:

[0453] The server analyzes the photos and videos.

[0454] How it works: The server's image analysis module uses OpenCV and TensorFlow to analyze photos and videos, recognizing scenery and people, and dividing videos into frames to extract movement.

[0455] Input: A saved photo or video file.

[0456] Output: Recognized scenery, people, and movement feature information.

[0457] Step 5:

[0458] The server generation AI generates a 3D metaverse space.

[0459] How it works: The extracted feature information is input into a generative AI, which then uses GAN and 3D reconstruction algorithms to generate a 3D metaverse space.

[0460] Input: Extracted feature information.

[0461] Output: The generated 3D metaverse spatial data.

[0462] Step 6:

[0463] Stores metaverse spatial data in a database.

[0464] How it works: The generated 3D Metaverse spatial data is stored in a database, and an identifier is generated and linked to the user's profile.

[0465] Input: Generated 3D metaverse spatial data.

[0466] Output: 3D metaverse spatial data and identifiers stored in a database.

[0467] Step 7:

[0468] A user accesses the metaverse space.

[0469] How it works: A user puts on a VR headset or AR-enabled device and logs into the server through a dedicated app. A list of saved Metaverse spaces is displayed, and the user can select a specific experience. The selected space data is streamed in real time.

[0470] Input: Login information to the server, identifier of the selected metaverse space.

[0471] Output: Metaverse spatial footage streamed in real time to the user's device.

[0472] Step 8:

[0473] The user's emotions are captured and the server tailors the experience.

[0474] How it works: The user's device captures facial expressions and voice and sends them to the emotion engine. The emotion engine analyzes the emotional data and sends it to the server. The server uses this data to adjust the visual and audio effects in the metaverse space in real time.

[0475] Input: Captured facial and voice data, analyzed emotion data.

[0476] Output: A metaverse spatial experience with visual and audio effects tuned in real time.

[0477] (Application example 2)

[0478] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0479] In modern society, taking photos and videos has become an important means of preserving and re-experiencing memories. However, it is difficult to fully reproduce the emotions and atmosphere of a moment with two-dimensional photos and videos, and they are unable to fully recreate the excitement and joy felt by the user. It is also difficult for a user to relive an experience captured at a specific location with a sense of presence from another location. Furthermore, no system has been available to date that can provide a customized experience based on the user's emotions. There is a need for a system that can solve these issues and provide a more immersive and emotionally responsive real-time experience.

[0480] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0481] In this invention, the server includes means for receiving photos or videos acquired from a user device and analyzing scenery, people, and movements from the photos or videos, means for a generation AI to generate a 3D metaverse space based on the analysis results, means for saving the generated 3D metaverse space and providing it for user access, means for acquiring emotional data from the user device, and means for adjusting visual and audio effects in the metaverse space in real time based on the acquired emotional data. This allows users to experience a highly immersive metaverse space based on the photos and videos they have taken, and allows them to enjoy a real-time experience customized to their emotions at the time.

[0482] A "user terminal" is a device that a user uses to take photos and videos and send the data to a server, and includes smartphones, tablets, digital cameras, etc.

[0483] "Server" refers to a computer system that receives and analyzes data sent from user terminals, and generates, stores, and provides the metaverse space.

[0484] "Generative AI" is a technology that includes deep learning models and 3D reconstruction algorithms for generating a three-dimensional metaverse space based on data received from user devices.

[0485] "Emotion data" is information about the emotional state obtained by analyzing the user's facial expressions and voice.

[0486] "Means for real-time adjustment" refers to technology that instantly adjusts visual and sound effects within the metaverse space based on acquired emotional data.

[0487] A "metaverse space" is a three-dimensional virtual space where users can experience virtual reality.

[0488] "Visual and audio effects" refers to visual and audio effects that enhance the user experience, such as adjusting lighting or changing background music.

[0489] This invention is a system that recreates the scenery, people, and movements of a moment in a three-dimensional metaverse space based on photos and videos taken by the user, and further adjusts the experience by recognizing the user's emotions. Detailed embodiments for carrying out this invention are described below.

[0490] First, a user terminal is a device that includes a smartphone, tablet, digital camera, etc., that allows the user to take photos and videos and send the data to a server. When a user takes a photo or video, the data is sent to the server through a dedicated application. At this stage, the user's facial expressions and voice are also captured and sent as emotion data.

[0491] The server receives photo, video, and emotion data sent from the user's device. It analyzes the received data and uses computer vision technology to extract scenery, people, and movements. Specifically, image analysis is performed using libraries such as OpenCV and TensorFlow. Based on the analysis results, a generative AI generates a 3D metaverse space. The generative AI includes deep learning models and 3D reconstruction algorithms.

[0492] The generated metaverse space is stored on a server and made accessible to users. Users can access the generated metaverse space and experience it with a high level of immersion using a virtual reality headset or an augmented reality-enabled device.

[0493] Furthermore, the emotion engine analyzes the user's facial expressions and voice to recognize their emotional state, and adjusts the visual and audio effects in the metaverse space in real time based on the recognized emotion. For example, if the user is emotional, the background music can be changed to a more uplifting one, or the lighting can be adjusted to be more vibrant.

[0494] Specific examples

[0495] Users take photos of new shoes and upload them to a server using a dedicated application. The server analyzes the received photos, generates a 3D model of the shoe, and stores it for users to access. Users can then put on a virtual reality headset and enter a virtual store to try on the generated 3D shoe model. The user's facial expressions and voice are analyzed by an emotion engine, which adjusts the lighting and background music in the store based on the emotion data.

[0496] Prompt Sentence Examples

[0497] "Generate a realistic 3D model based on a photograph of a new running shoe. The model should accurately reproduce the shoe's details and specific features (e.g., sole design, color tone)."

[0498] This allows users to experience a highly immersive metaverse space based on the photos and videos they take, and allows them to enjoy a real-time experience customized to their emotions at the time.

[0499] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0500] Step 1:

[0501] The user takes a photo or video.

[0502] Input: Photos and videos taken by users using devices such as smartphones and tablets.

[0503] Specific operation: The user launches the photo app and takes photos or videos while traveling or at an event. The captured data is saved in the device's local storage.

[0504] Output: Photo and video data saved in local storage.

[0505] Step 2:

[0506] The user uploads data.

[0507] Input: Photo and video data stored on the user's device.

[0508] How it works: The user opens the application and selects the photos and videos they have taken for uploading. The selected data is sent to the server as an HTTP POST request.

[0509] Output: Photo and video data sent to the server.

[0510] Step 3:

[0511] The server receives the data.

[0512] Input: HTTP POST request sent from the user's device.

[0513] Specific operation: The server receives the HTTP request and temporarily saves the sent photo or video files in the specified directory.

[0514] Output: Temporarily saved photo and video files.

[0515] Step 4:

[0516] The server analyzes the photos and videos.

[0517] Input: Temporarily saved photo and video files.

[0518] How it works: The server's image analysis module analyzes photo and video data using computer vision technologies such as OpenCV and TensorFlow, recognizing scenery, people, and movements, and extracting features from each frame.

[0519] Output: Analyzed feature information (scenery, people, movement) data.

[0520] Step 5:

[0521] The server uses generation AI to generate a three-dimensional metaverse space.

[0522] Input: Parsed feature information data.

[0523] Specific operation: The server uses generative AI, including deep learning models and 3D reconstruction algorithms, to generate a three-dimensional metaverse space based on the analysis results.

[0524] Output: Generated 3D metaverse spatial data.

[0525] Step 6:

[0526] The server stores and provides the generated three-dimensional metaverse space.

[0527] Input: Generated 3D Metaverse spatial data.

[0528] Specific operation: The server stores the generated 3D metaverse space data in a database, links the identifier to the user's profile, and provides the stored data for the user to access.

[0529] Output: Metaverse spatial data stored in a database.

[0530] Step 7:

[0531] Emotion data is acquired from the user's device.

[0532] Input: Facial expressions and voice captured by the user's device.

[0533] Specific operation: The user device captures facial expressions and voices during the user's experience and sends them to the emotion engine.

[0534] Output: Facial and speech data captured and sent to the emotion engine.

[0535] Step 8:

[0536] Adjusting the metaverse space in real time based on emotional data.

[0537] Input: Acquired emotion data.

[0538] Specific operation: The server analyzes the data sent from the emotion engine, recognizes the user's emotional state, and instantly adjusts the visual and sound effects in the metaverse space based on the recognized emotion.

[0539] Output: A real-time adjusted metaverse space.

[0540] Step 9:

[0541] A user accesses the metaverse space.

[0542] Input: User login information and metaverse space identifier.

[0543] Specific operation: The user puts on a head-mounted display or augmented reality device and launches a dedicated application. After logging in, the user selects a specific space from a list of saved metaverse spaces to begin the experience. The selected metaverse space data is then streamed in real time from the server and displayed to the user.

[0544] Output: A real-time display of the metaverse space experienced by the user.

[0545] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0546] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search<url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0547] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0548] [Second embodiment]

[0549] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0550] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0551] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0552] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0553] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0554] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0555] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0556] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0557] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0558] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0559] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0560] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0561] This invention is a system that recreates the scenery, people, and movements of a moment in a metaverse space based on photos and videos taken by the user. This system utilizes the user's device, a server, and generation AI to provide the user with an immersive experience.

[0562] System configuration

[0563] 1. User Device:

[0564] They act as devices that allow users to take photos and videos, including smartphones, tablets, and digital cameras.

[0565] A dedicated application is installed to allow users to send captured image data to a server.

[0566] Users access the saved metaverse space and use VR headsets or AR-enabled devices to experience it.

[0567] 2. Server:

[0568] It has a component that receives photo and video data sent from the user's device.

[0569] Computer vision techniques are used to analyze the received data and extract scenery, people, and movement.

[0570] Based on the extracted information, a 3D metaverse space is generated using generative AI.

[0571] The generated metaverse spatial data is stored in a database and provided for users to access.

[0572] 3. Generation AI:

[0573] Based on the analysis results, it includes deep learning models and 3D reconstruction algorithms to generate 3D models that reproduce real-world scenes, people, and movements.

[0574] System Operation

[0575] Data acquisition and transmission (user's device)

[0576] User takes a photo or video:

[0577] A user takes photos and videos using the device's camera app while traveling or at an event.

[0578] The captured data is stored in the device's local storage.

[0579] User uploads data:

[0580] The user opens the dedicated app and goes to the upload screen of the metaverse generation service.

[0581] Select the saved photo or video file and press the upload button.

[0582] The app sends the selected file to the server as an HTTP POST request.

[0583] Data reception and analysis (server)

[0584] Server receives data:

[0585] The server receives the HTTP request and retrieves the transmitted file data.

[0586] The received file is stored in a directory for temporary storage.

[0587] The server analyzes the photos and videos:

[0588] The server's image analysis module analyzes the transmitted photo and video data.

[0589] Image analysis uses computer vision libraries such as OpenCV and TensorFlow to recognize scenery and people.

[0590] In video analysis, a video is divided into frames, and each frame is subjected to image analysis to extract movement.

[0591] Generation and storage of 3D metaverse space (server)

[0592] The server generates a 3D metaverse space using generative AI:

[0593] The extracted feature information is input into a generation AI to generate a 3D metaverse space.

[0594] Generative AI uses GANs and 3D reconstruction algorithms to recreate realistic situations.

[0595] The server saves the generated metaverse space:

[0596] The generated 3D data is stored in a database along with metadata.

[0597] Generate an identifier (ID) for the stored spatial data and link it to the user's profile.

[0598] Providing a metaverse experience (user device)

[0599] User accesses the metaverse space:

[0600] The user puts on a VR headset or AR-compatible device and launches a dedicated app.

[0601] Log in to the server from the app and view a list of saved metaverse spaces.

[0602] Select a specific metaverse space from the list provided to begin the experience.

[0603] The app streams selected metaverse spatial data from the server and displays it to the user in real time.

[0604] Specific examples

[0605] 1. Relive your travel memories:

[0606] A user takes a photo or video in front of the Eiffel Tower while traveling in Paris (user's device).

[0607] The taken photo is uploaded by dragging and dropping it onto the upload screen of the metaverse generation service (user's device).

[0608] The server receives the photo data sent from the user's terminal and temporarily stores it (server).

[0609] The server's image analysis module recognizes the Eiffel Tower and the surrounding scenery from the sent photo and extracts its features (server).

[0610] The generative AI generates a 3D model of the Eiffel Tower based on the extracted features and recreates the background scenery (server).

[0611] The generated 3D model and scenery data are stored in a database and linked to the user's profile (server).

[0612] Users put on a VR headset at home and open a dedicated app to access the Eiffel Tower metaverse space (on their device).

[0613] Users can move freely within the metaverse and experience the Eiffel Tower and its surroundings in 360 degrees (on their device).

[0614] 2. Recreating child growth records:

[0615] A user records a video of their child's first birthday (on their device).

[0616] The captured video is uploaded to the metaverse generation service (user's device).

[0617] The server receives the video and analyzes the birthday cake and the child's movements (server).

[0618] Based on the analyzed information, the generation AI generates a 3D metaverse space for the birthday party (server).

[0619] The generated 3D data is stored in a database and made accessible to users (server).

[0620] The user uses a VR headset to relive their child's first birthday (on the user's device).

[0621] The system of the present invention allows users to realistically preserve and experience memories in a highly immersive alternative to two-dimensional photos and videos.

[0622] The processing flow will be explained below.

[0623] Step 1:

[0624] Data acquisition (user device)

[0625] Users use the camera app on their devices to take photos or videos of scenery at their travel destinations or events.

[0626] The captured data is stored in the device's local storage.

[0627] Step 2:

[0628] Data transmission (user's device)

[0629] Users open the dedicated app and select the photos and videos they have taken.

[0630] When you press the upload button, the app sends the selected data to the server as an HTTP POST request.

[0631] Step 3:

[0632] Data reception (server)

[0633] The server receives an HTTP POST request to retrieve the photo or video file.

[0634] To temporarily store the received data, save it in the specified directory.

[0635] Step 4:

[0636] Image and video analysis (server)

[0637] The server uses computer vision libraries (e.g., OpenCV and TensorFlow) to analyze the stored photo and video data.

[0638] In the case of images, the server extracts features such as scenery, people, and objects from the image.

[0639] In the case of video, the server divides the video into frames, analyzes each frame, and extracts movement.

[0640] Step 5:

[0641] 3D Metaverse Space Generation (Server)

[0642] Based on the analysis results, the server generates a 3D metaverse space using generative AI (e.g., GAN or 3D reconstruction algorithms).

[0643] Generative AI uses extracted features to recreate scenes, people, and movements.

[0644] Step 6:

[0645] Metaverse space storage (server)

[0646] The server stores the generated 3D metaverse spatial data in a database.

[0647] Generate an identifier (ID) for the saved space and link it to the user's profile.

[0648] Step 7:

[0649] Providing a metaverse experience (user device)

[0650] The user puts on a VR headset or AR-compatible device and launches a dedicated app.

[0651] Users log in to the server through the app and view a list of saved metaverse spaces.

[0652] Select a specific metaverse space from the list provided to begin the experience.

[0653] The app streams selected metaverse spatial data in real time from the server and displays it to the user.

[0654] In this way, users can experience an immersive three-dimensional metaverse space generated from two-dimensional photos and videos.

[0655] Example 1

[0656] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0657] Conventional recording methods using photographs and videos forced users to rely on two-dimensional information, making it difficult to recreate the atmosphere or three-dimensional experience of the scene. Furthermore, there were limited ways to preserve detailed memories of specific events or travel destinations and experience them later in real life. Furthermore, existing systems required complex processes for data analysis and reproduction, making practical application difficult.

[0658] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0659] In this invention, the server includes: means for receiving photos or videos acquired from a user terminal and analyzing scenery, people, and movements from the photos or videos; means for a generation AI to generate a 3D metaverse space based on the analysis results; means for saving the generated 3D metaverse space and providing it for user access; means for a user to take photos or videos and upload the data to the server using a dedicated application; means for the server to save the transmitted photos or videos in a temporary directory and analyze them using computer vision technology; and means for saving the generated metaverse space data in a database and linking the identifier to a user profile. This allows users to have a more immersive experience, enabling the realistic reproduction of on-site atmospheres and three-dimensional experiences.

[0660] "User terminal" refers to any digital device that allows a user to take and save photos and videos, and specific examples include smartphones and digital cameras.

[0661] A "server" refers to a computer system that receives, analyzes, and stores data sent from a user terminal, and provides various services to the user.

[0662] "Photos or videos" refers to still images or video data taken by a user using a digital device.

[0663] "Analysis" refers to the process of recognizing scenery, people, and movements from photos or videos and extracting their respective characteristics.

[0664] "Scenery" refers to the natural or man-made scenery or backgrounds contained in photographs or videos.

[0665] "People" refers to human figures appearing in photographs and videos.

[0666] "Motion" refers to the movement and activity of people and objects in a photo or video.

[0667] "Generative AI" refers to artificial intelligence techniques used to generate 3D Metaverse spaces based on analyzed data, including GANs and 3D reconstruction algorithms.

[0668] "3D Metaverse space" refers to a digital space in which real-world scenery, people, and movements are reproduced in three dimensions within a virtual space.

[0669] "Dedicated application" refers to software that users use to upload photos and videos to a server.

[0670] The "temporary directory" refers to a folder where the server temporarily stores data received.

[0671] "Computer vision technology" refers to the technology that allows computers to analyze images and videos and extract information. Examples include OpenCV and TensorFlow.

[0672] "Database" refers to a system for efficiently storing and managing the generated metaverse spatial data. A specific example is MongoDB.

[0673] "Identifier" refers to an ID that uniquely identifies stored metaverse spatial data.

[0674] "User profile" refers to a digital data structure for storing information and data related to a user.

[0675] "VR headset" refers to a device used by a user to experience a virtual reality space, including, for example, Oculus Rift.

[0676] "AR-enabled device" means a device used to experience an augmented reality space, including, for example, Microsoft HoloLens.

[0677] MODE FOR CARRYING OUT THE INVENTION

[0678] This invention is a system that recreates the scenery, people, and movements of a moment in a metaverse space based on photos and videos taken by the user. This system utilizes the user's device, a server, and generation AI to provide the user with an immersive experience.

[0679] 1. System Configuration

[0680] User terminal

[0681] A user device is a general term for a digital device that allows a user to take and save photos and videos, including smartphones and digital cameras. A dedicated application (e.g., the Metaverse Creator app) is installed on the device to transmit the captured data to a server. Users can access and experience the saved Metaverse space using a VR headset (e.g., Oculus Rift) or an AR-enabled device (e.g., Microsoft HoloLens).

[0682] server

[0683] The server receives photo and video data sent from user devices, analyzes it, and generates the metaverse space. The server includes the following main components:

[0684] Data receiving module: Receives HTTP requests and retrieves the transmitted file data.

[0685] Temporary storage: The directory where received data is temporarily stored (e.g. / tmp / uploads).

[0686] Image analysis module: Uses computer vision libraries such as OpenCV and TensorFlow to extract scenery, people, and movement from photo and video data.

[0687] Generative AI module: Uses GAN and 3D reconstruction algorithms to generate a 3D metaverse space based on extracted data.

[0688] Database: A database (e.g., MongoDB) to store the generated metaverse spatial data and associated metadata.

[0689] 2. System Operation

[0690] Data acquisition and transmission by the user's device

[0691] A user takes photos and videos using a smartphone or digital camera. For example, while traveling, the user takes a photo of the Eiffel Tower using the iPhone camera app and saves it to the camera roll. The user then launches the MetaverseCreator app, selects the photo, and uploads it. The app then sends the photo data to the server as an HTTP POST request.

[0692] Data reception and analysis by the server

[0693] The server receives the HTTP request and saves the data in a temporary directory. The image analysis module then analyzes the photo data, recognizing scenery such as the Eiffel Tower, people, and movements, and extracting feature points. This feature point information is stored as JSON data.

[0694] Creation and storage of 3D metaverse space

[0695] The server inputs the extracted feature information into a generative AI module, which then uses a GAN model and 3D reconstruction algorithm to generate a 3D structure of the Eiffel Tower and recreate the background scenery. The generated metaverse spatial data is stored in a database, and an identifier is generated and linked to the user's profile.

[0696] User experience of the metaverse space

[0697] Users put on a VR headset or AR-enabled device and launch the MetaverseViewer app. They log in within the app and view a list of Metaverse spaces linked to their profile. They select a specific Metaverse space and begin the experience. The app streams the Metaverse space data from the server and displays it on the user's device in real time.

[0698] Examples and prompts

[0699] For example, if a user uploads a photo of the Eiffel Tower taken during a trip to Paris, and a generative AI generates a 3D metaverse of the Eiffel Tower based on that photo, the actual prompt would look like this:

[0700] "Generate a 3D model of the Eiffel Tower based on the following photo."

[0701] "Reconstruct a scene from a child's birthday party using frames from this video."

[0702] The system of the present invention allows users to realistically preserve and experience memories in a highly immersive alternative to two-dimensional photos and videos.

[0703] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0704] Step 1:

[0705] A user takes a photo or video. A user takes a photo or video using a smartphone or digital camera. For example, while traveling, a user takes a photo of the Eiffel Tower using the camera app on their iPhone and saves it to their camera roll. The input is the camera app on the digital device, and the output is the photo or video data saved in local storage.

[0706] Step 2:

[0707] The user uploads data to the server using a dedicated application. The user launches the MetaverseCreator app, selects the photos and video files they have taken, and uploads them. The input is the photo or video file saved in local storage, and the output is the file data sent as an HTTP POST request. During this process, an upload progress bar is displayed.

[0708] Step 3:

[0709] The server receives the data. The server receives the HTTP request and saves the submitted file data and metadata in a temporary directory (e.g., / tmp / uploads). The input is the photo or video data included in the HTTP request body, and the output is the file saved in the temporary directory.

[0710] Step 4:

[0711] The server analyzes photos and videos. The server's image analysis module (e.g., OpenCV, TensorFlow) analyzes the photo and video data stored in a temporary directory. The input is the file stored in the temporary directory, and the output is feature information about the scenery, people, and actions stored in JSON format. Specifically, feature points of the Eiffel Tower and people are extracted from photos, and each frame of the video is analyzed to detect movement.

[0712] Step 5:

[0713] The server uses generative AI to generate a 3D metaverse space. The server inputs the extracted feature point information into a generative AI module (e.g., GAN or 3D reconstruction algorithm) to generate the 3D metaverse space. The input is JSON data of the feature point information, and the output is data in the generated 3D metaverse space. For example, a GAN model generates the 3D structure of the Eiffel Tower and recreates the background scenery.

[0714] Step 6:

[0715] The server saves the generated metaverse space in a database. The server saves the generated metaverse space data in a database (e.g., MongoDB) and links the identifier to the user profile. The input is the 3D metaverse space data and the user ID, and the output is the data saved in the database and the generated identifier.

[0716] Step 7:

[0717] A user accesses the Metaverse space by putting on a VR headset (e.g., Oculus Rift) or an AR-enabled device (e.g., Microsoft HoloLens) and launching the MetaverseViewer app. The input is the application on the device, and the output is the Metaverse space data streamed from the server. When the user selects a Metaverse space in the app and presses the "Start Experience" button, the app displays the data in real time, and the user enjoys an immersive experience.

[0718] (Application example 1)

[0719] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0720] There is a need for a method that allows users to upload photos and videos they have taken to the metaverse space through a dedicated application, and then share the 3D metaverse space generated from it with other users, thereby providing a visually immersive experience. However, existing technologies do not easily enable the generation and sharing of such advanced metaverse spaces, so a new solution is needed.

[0721] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0722] In this invention, the server includes means for receiving photos or videos acquired from a user terminal and analyzing scenery, people, and movements from the photos or videos, means for a generation AI to generate a 3D metaverse space based on the analysis results, means for saving the generated 3D metaverse space and providing it for users to access, means for users to upload and share photos or videos using a smartphone application, and means for sharing the generated 3D metaverse space with friends and family. This allows users to generate realistic 3D metaverse spaces based on photos and videos they have taken and easily share them with other users, providing a visually realistic and immersive experience.

[0723] "User terminal" refers to a device that takes photos and videos and has installed an application for sending that data to a server.

[0724] A "server" refers to a computer system that receives data sent from a user terminal, analyzes it, and performs the necessary processing.

[0725] "Generative AI" refers to artificial intelligence technology for generating a three-dimensional metaverse space based on the analysis of photos and videos sent from user devices.

[0726] "3D Metaverse space" refers to a three-dimensional virtual space generated based on real-world scenery, people, and movements.

[0727] "Smartphone application" refers to software that is installed on a smartphone and allows users to upload photos and videos, share and experience the metaverse space.

[0728] "Analyzing" refers to the process of extracting specific information (scenery, people, movements, etc.) from photos and videos.

[0729] "Saving" refers to the act of storing the generated 3D metaverse spatial data in a state that allows it to be accessed later.

[0730] "Providing" refers to the process of allowing a user to access a saved three-dimensional metaverse space.

[0731] "Sharing" refers to the act of exchanging the generated 3D metaverse space with other users in an easily accessible manner.

[0732] "Friends and Family" refers to close people with whom the user wants to share.

[0733] This invention is a system that recreates the scenery, people, and movements of a moment in a 3D metaverse space based on photos and videos taken by the user. This system utilizes the user's device, a server, and a generation AI to provide the user with an immersive experience.

[0734] System configuration

[0735] It consists of the following main components:

[0736] 1. User Device:

[0737] Camera device: A device that allows a user to take photos or videos. Examples include smartphones, tablets, and digital cameras.

[0738] Dedicated application: An application is installed to send the captured data to the server. VR headsets and AR-compatible devices are also used to access and experience the generated 3D metaverse space.

[0739] 2. Server:

[0740] Receiving module: Contains a component that receives and temporarily stores photo and video data sent from the user device.

[0741] Image analysis module: Analyzes received photo and video data using computer vision technology, specifically using libraries such as OpenCV and TensorFlow.

[0742] Generative AI module: Generates a 3D metaverse space based on the extracted information. This uses deep learning models, including GANs and 3D reconstruction algorithms.

[0743] Database: Stores the generated 3D metaverse spatial data and makes it accessible to users.

[0744] 3. Generation AI:

[0745] Deep learning model: A technology for generating 3D models that reproduce real-world scenery, people, and movements based on analysis results sent from the user's device.

[0746] 3D reconstruction algorithm: Creates a realistic 3D space based on the analysis results.

[0747] System Operation

[0748] 1. Data acquisition and transmission (user terminal):

[0749] Users take photos and videos using their smartphones and upload the data to the server via a dedicated application. Uploads are made via HTTP POST requests.

[0750] 2. Data reception and analysis (server):

[0751] The server receives and temporarily stores the photos and video data sent from the user's device. It then uses computer vision technology to analyze the scenery, people, and movements from the received data. Libraries such as OpenCV and TensorFlow are used for this.

[0752] 3. Generation and storage of 3D metaverse space (server):

[0753] The analysis results are input into the generation AI to generate a 3D metaverse space. The generated 3D data is stored in a database and made available for users to access.

[0754] 4. Providing a Metaverse experience (user device):

[0755] Users can access the generated 3D metaverse space through a dedicated application using a VR headset or AR-compatible device, and can also share the generated metaverse space with friends and family.

[0756] Specific examples

[0757] Travel Reenactment:

[0758] Users upload photos taken in front of the Eiffel Tower during their travels. The server analyzes the photos and uses generative AI to generate a 3D metaverse of the Eiffel Tower and its surroundings. Users can access this metaverse by wearing a VR headset and enjoy a 360-degree experience.

[0759] Recreating a child's growth record:

[0760] Users upload videos of their child's first birthday. The server analyzes the videos and uses generative AI to generate a 3D space of the birthday party. Users can access this recreated space and relive their child's growth and memories.

[0761] Prompt Sentence Examples

[0762] "Upload a photo of the Eiffel Tower and generate a realistic 3D metaverse space based on this photo."

[0763] This system allows users to easily recreate their own photos and videos in a 3D metaverse space and share that space with others, providing a visually immersive experience.

[0764] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0765] Step 1:

[0766] The user takes a photo or video

[0767] Users take photos and videos using the camera app on their smartphones.

[0768] Input: Photo or video data from a smartphone camera app

[0769] Output: Photo or video files stored in the smartphone's local storage

[0770] Step 2:

[0771] User uploads data

[0772] The user opens the application, selects the photos and video files they have taken, and presses the upload button, which sends these files to the server as an HTTP POST request.

[0773] Input: Photo or video files selected through a dedicated application, server upload endpoint

[0774] Output: HTTP POST request sent to the server

[0775] Step 3:

[0776] The server receives the data

[0777] The server receives the HTTP POST request, retrieves the sent file data, and saves it in a temporary directory.

[0778] Input: Photo or video data received via an HTTP POST request

[0779] Output: File data saved in temporary directory

[0780] Step 4:

[0781] The server analyzes the photos and videos.

[0782] The server's image analysis module uses OpenCV and TensorFlow to analyze the received photo and video data. Image analysis recognizes scenery and people, while video analysis divides the video into frames to extract movement.

[0783] Input: Photo or video data saved in the temporary directory

[0784] Output: Scenery, people, and movement information as analysis results

[0785] Step 5:

[0786] The server generates a 3D metaverse space using generation AI.

[0787] The server inputs the analysis results into the generation AI, which then generates a 3D metaverse space using GAN and 3D reconstruction algorithms.

[0788] Input: Scenery, people, and movement information as analysis results

[0789] Output: Generated 3D metaverse spatial data

[0790] Step 6:

[0791] The server stores the generated 3D metaverse spatial data.

[0792] The server stores the generated 3D metaverse spatial data in a database and makes it accessible to users.

[0793] Input: Generated 3D metaverse spatial data, user identification information

[0794] Output: 3D metaverse spatial data stored in a database, metaverse space linked to user profile

[0795] Step 7:

[0796] A user accesses the metaverse space

[0797] Users use a VR headset or AR-compatible device to log in through a dedicated application, select one from a list of saved 3D metaverse spaces, and begin the experience.

[0798] Input: User login information, list of 3D metaverse spaces stored in the database

[0799] Output: 3D metaverse space streamed to user's device

[0800] Step 8:

[0801] Users share the metaverse space

[0802] Users can share the 3D metaverse space they create with friends and family through a dedicated application, generating a shared link and granting access rights to specific users.

[0803] Input: 3D metaverse space linked to user profile, shared identity information

[0804] Output: Shared link and access rights to the shared 3D metaverse space

[0805] The above are the specific processing steps in the system for implementing this invention. At each step, various data processing and calculations are performed based on the input data, and the results are passed on to the next step, thereby operating the entire system.

[0806] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0807] The present invention is a system that recreates the scenery, people, and movements of a user in a metaverse space based on photos and videos taken by the user, and further adjusts the experience by recognizing the user's emotions. The following describes the detailed operation of the system.

[0808] System configuration

[0809] 1. User Device:

[0810] They act as devices that allow users to take photos and videos, including smartphones, tablets, and digital cameras.

[0811] A dedicated application is installed to allow users to send captured image data to a server.

[0812] It also has the function of capturing the user's facial expressions and voice and sending them to the emotion engine.

[0813] Users access the saved metaverse space and use VR headsets or AR-enabled devices to experience it.

[0814] 2. Server:

[0815] It has a component that receives photo and video data sent from the user's device, as well as emotion data recognized by the emotion engine.

[0816] Computer vision techniques are used to analyze the received data and extract scenery, people, and movement.

[0817] Based on the extracted information, a 3D metaverse space is generated using generative AI.

[0818] The generated metaverse spatial data is stored in a database and provided for users to access.

[0819] Tailor your metaverse experience in real time based on data from the emotion engine.

[0820] 3. Generation AI:

[0821] Based on the analysis results, it includes deep learning models and 3D reconstruction algorithms to generate 3D models that reproduce real-world scenes, people, and movements.

[0822] 4. Emotion Engine:

[0823] Recognizes emotions by analyzing the user's facial expressions and voice.

[0824] Adjust visual and audio effects within the metaverse space based on the emotions recognized.

[0825] System Operation

[0826] Data acquisition and transmission (user's device)

[0827] User takes a photo or video:

[0828] A user takes photos and videos using the device's camera app while traveling or at an event.

[0829] The captured data is stored in the device's local storage.

[0830] User uploads data:

[0831] The user opens the dedicated app and selects the photos and videos they have taken.

[0832] When you press the upload button, the app sends the selected data to the server as an HTTP POST request.

[0833] Data reception and analysis (server)

[0834] Server receives data:

[0835] The server receives an HTTP POST request to retrieve the photo or video file.

[0836] To temporarily store the received data, save it in the specified directory.

[0837] The server analyzes the photos and videos:

[0838] The server's image analysis module analyzes the transmitted photo and video data.

[0839] Image analysis uses computer vision libraries such as OpenCV and TensorFlow to recognize scenery and people.

[0840] In video analysis, a video is divided into frames, and each frame is subjected to image analysis to extract movement.

[0841] 3D Metaverse Space Generation (Server)

[0842] The server generates a 3D metaverse space using generative AI:

[0843] The extracted feature information is input into a generation AI to generate a 3D metaverse space.

[0844] Generative AI uses GANs and 3D reconstruction algorithms to recreate realistic situations.

[0845] Metaverse space storage (server)

[0846] The server stores the generated 3D metaverse space:

[0847] The server stores the generated 3D metaverse spatial data in a database.

[0848] Generate an identifier (ID) for the stored spatial data and link it to the user's profile.

[0849] Emotion recognition and user experience adjustment (emotion engine and server)

[0850] Emotion engine recognizes emotions:

[0851] The user device captures the user's facial expressions and voice and sends them to the emotion engine.

[0852] The emotion engine analyzes this data and recognizes the user's emotional state.

[0853] Emotion-based experience tailoring:

[0854] The emotion data obtained from the emotion engine is received by the server.

[0855] The server changes the visual and sound effects in the metaverse space in real time based on the emotional data.

[0856] For example, if the user is impressed, the scenery will be made more vivid and the background music will be changed to something more exciting.

[0857] Providing a metaverse experience (user device)

[0858] User accesses the metaverse space:

[0859] The user puts on a VR headset or AR-compatible device and launches a dedicated app.

[0860] Users log in to the server through the app and view a list of saved metaverse spaces.

[0861] Select a specific metaverse space from the list provided to begin the experience.

[0862] The app streams selected metaverse spatial data in real time from the server and displays it to the user.

[0863] Specific examples

[0864] 1. Travel memory recall and emotional regulation:

[0865] A user takes a photo or video in front of the Eiffel Tower while traveling in Paris (user's device).

[0866] The taken photo is uploaded by dragging and dropping it onto the upload screen of the metaverse generation service (user's device).

[0867] The server receives the photo data sent from the user's device and temporarily stores it (server).

[0868] The server's image analysis module recognizes the Eiffel Tower and the surrounding scenery from the sent photo and extracts its features (server).

[0869] The generative AI generates a 3D model of the Eiffel Tower based on the extracted features and recreates the background scenery (server).

[0870] The generated 3D model and scenery data are stored in a database and linked to the user's profile (server).

[0871] Users put on a VR headset at home and open a dedicated app to access the Eiffel Tower metaverse space (on their device).

[0872] While the user is experiencing the experience, the emotion engine analyzes the user's facial expressions and if it recognizes emotion, the server adjusts the effects and changes the background music to an emotional one (server).

[0873] 2. Recapitulation of child development records and emotional regulation:

[0874] A user records a video of their child's first birthday (on their device).

[0875] The captured video is uploaded to the metaverse generation service (user's device).

[0876] The server receives the video and analyzes the birthday cake and the child's movements (server).

[0877] Based on the analyzed information, the generation AI generates a 3D metaverse space for the birthday party (server).

[0878] The generated 3D data is stored in a database and made accessible to users (server).

[0879] The user uses a VR headset to relive their child's first birthday (on the user's device).

[0880] If the emotion engine recognizes the user's joy, the server will brighten the lights in the space and make the background music more enjoyable (server).

[0881] The system of the present invention allows users to realistically preserve memories in a highly immersive alternative to two-dimensional photos and videos, and allows for customized experiences that reflect their emotional state at any given time.

[0882] The processing flow will be explained below.

[0883] Step 1:

[0884] Data acquisition (user device)

[0885] Users use the camera app on their devices to take photos or videos of scenery at their travel destinations or events.

[0886] The captured data is stored in the device's local storage.

[0887] Step 2:

[0888] Data transmission (user's device)

[0889] Users open the dedicated app and select the photos and videos they have taken.

[0890] When you press the upload button, the app sends the selected data to the server as an HTTP POST request.

[0891] Step 3:

[0892] Data reception (server)

[0893] The server receives an HTTP POST request to retrieve the photo or video file.

[0894] To temporarily store the received data, save it in the specified directory.

[0895] Step 4:

[0896] Image and video analysis (server)

[0897] The server uses computer vision libraries (e.g., OpenCV and TensorFlow) to analyze the stored photo and video data.

[0898] In the case of images, the server extracts features such as scenery, people, and objects from the image.

[0899] In the case of video, the server divides the video into frames, analyzes each frame, and extracts movement.

[0900] Step 5:

[0901] 3D Metaverse Space Generation (Server)

[0902] Based on the analysis results, the server generates a 3D metaverse space using generative AI (e.g., GAN or 3D reconstruction algorithms).

[0903] Generative AI uses extracted features to recreate scenes, people, and movements.

[0904] Step 6:

[0905] Metaverse space storage (server)

[0906] The server stores the generated 3D metaverse spatial data in a database.

[0907] Generate an identifier (ID) for the saved space and link it to the user's profile.

[0908] Step 7:

[0909] Data acquisition for emotion recognition (user device)

[0910] The user puts on a VR headset or AR-compatible device and launches the dedicated app.

[0911] Sensors and microphones begin to operate to capture the user's facial expressions and voice.

[0912] Step 8:

[0913] Sending emotion data (user device)

[0914] The user's facial expression and voice data are sent to the server in real time.

[0915] This data is then forwarded to a dedicated endpoint to be analyzed by the emotion engine.

[0916] Step 9:

[0917] Receiving and analyzing emotion data (server and emotion engine)

[0918] The server passes the received facial expression and voice data to the emotion engine.

[0919] The emotion engine analyzes this data and recognizes the user's emotional state.

[0920] Step 10:

[0921] Emotion-based experience adjustment (server)

[0922] The server adjusts the visual and sound effects in the metaverse space in real time based on the emotional state data obtained from the emotion engine.

[0923] For example, if the user is moved, the scenery is made more vivid and the background music is changed to something more moving.

[0924] Step 11:

[0925] Providing a metaverse experience (user device)

[0926] Users log in to the server through a dedicated app and view a list of saved metaverse spaces.

[0927] Select a specific metaverse space from the list provided to begin the experience.

[0928] The app streams selected metaverse spatial data in real time from the server and displays it to the user.

[0929] In this way, users can experience an immersive three-dimensional metaverse space generated from two-dimensional photos and videos, and further customize it to suit their emotions at the time.

[0930] Example 2

[0931] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0932] Conventional methods simply record photos and videos, but have the problem of not being able to reflect or reproduce the user's emotions and experiences in real time. Furthermore, simple 2D data lacks a sense of immersion, making it difficult to reproduce the emotions and joy that users actually felt at the time.

[0933] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0934] In this invention, the server includes means for receiving photos or videos acquired from a user terminal and analyzing scenery, people, and movements from the photos or videos, means for a generation AI to generate a 3D metaverse space based on the analysis results, means for saving the generated 3D metaverse space and providing it for user access, means for capturing the user's facial expressions and voice and transmitting them to an emotion engine that recognizes emotions, and means for adjusting visual and audio effects in the metaverse space in real time based on the recognized emotion data. This allows users to realistically recreate past experiences with a high level of immersion and have a customized experience that matches their emotions at the time.

[0935] "User terminal" refers to a device that a user uses to take photos and videos and upload the data to a server, and includes smartphones, tablets, digital cameras, etc.

[0936] "Server" refers to a computer system that receives data sent from a user terminal, analyzes, processes, stores, and provides the data.

[0937] "Photos or videos" refers to still images or video files taken by a user using a device.

[0938] "Analysis of scenery, people, and movements" refers to the process of using computer vision technology to recognize scenery, people, and movements in photographs or video data and extract their characteristics.

[0939] "Generative AI" refers to deep learning models and 3D reconstruction algorithms for generating a 3D metaverse space based on extracted feature information.

[0940] "3D Metaverse space" refers to a three-dimensional virtual space generated by a generation AI to recreate real-world situations as virtual environments.

[0941] An "emotion engine" refers to software or hardware that analyzes a user's facial expressions and voice and recognizes the user's emotional state.

[0942] "Real-time adjustment of visual and audio effects" refers to the process of dynamically changing the images and audio displayed within the metaverse space to match the user's emotions based on recognized user emotional data.

[0943] "VR headset" refers to a device worn on the head to immerse a user in a virtual reality environment.

[0944] "AR-compatible device" refers to a device that uses augmented reality technology to overlay virtual objects on the real world.

[0945] "Computer vision technology" refers to technology that enables computers to analyze and understand the content of images and videos, and includes libraries such as OpenCV and TensorFlow.

[0946] This invention is a system that recreates the scenery, people, and movements of a moment in a 3D metaverse space based on photos and videos taken on a user's device, and adjusts the experience by recognizing the user's emotions. The detailed operation of the system is explained below.

[0947] A user takes a photo or video using a user device such as a smartphone, tablet, or digital camera. This data is temporarily stored on the user's device. The user then uses a dedicated application to select the photos or videos they have taken and upload them to the server. Uploads are made as HTTP POST requests.

[0948] The server temporarily stores the received photo or video data in a specified directory. The server's image analysis module then analyzes the data to recognize and extract scenery, people, and movements. This analysis uses computer vision technologies such as OpenCV and TensorFlow.

[0949] Based on the analysis results, the server uses generative AI to generate a 3D metaverse space. This generative AI uses GAN (generative adversarial networks) and 3D reconstruction algorithms to create a realistic virtual space. The generated 3D metaverse space data is then stored in a database, and an identifier is generated and linked to the user's profile.

[0950] Users put on a VR headset or AR-compatible device and log in to the server through a dedicated application. A list of saved 3D metaverse spaces is displayed, and the user selects a specific space to begin the experience. The selected space data is streamed in real time by the server and displayed on the user's device.

[0951] During the experience, the user's device captures facial expressions and voice and sends this data to the emotion engine. The emotion engine analyzes the user's emotional state and sends the recognized emotional data to the server. The server uses this data to adjust the visual and audio effects in the metaverse space in real time. For example, if the user is emotional, the scenery will be made more vivid and the background music will be more exciting.

[0952] For example, a user takes a photo in front of the Eiffel Tower while traveling in Paris and uploads the photo data to a server using a dedicated app. Based on this, the server generates a 3D metaverse space, which the user can access. The emotion engine recognizes the user's emotions during the experience, and the server adjusts the background music accordingly.

[0953] An example of a prompt sentence is as follows:

[0954] "Generate and display a 3D metaverse of the Eiffel Tower based on photos from a trip to Paris."

[0955] In this way, the system of the present invention allows users to realistically preserve memories in a highly immersive alternative to two-dimensional photos and videos, and allows for customized experiences based on emotional state.

[0956] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0957] Step 1:

[0958] The user takes a photo or video.

[0959] Action: The user takes a photo using a smartphone or digital camera.

[0960] Input: Real-world scenes and people.

[0961] Output: Captured photo or video files (formats such as JPEG or MP4).

[0962] Step 2:

[0963] The user uploads the captured data to the server.

[0964] How it works: Use the dedicated app to select the shooting data and press the upload button.

[0965] Input: The captured photo or video file.

[0966] Output: Sent as an HTTP POST request to the server.

[0967] Step 3:

[0968] The server receives and stores the data.

[0969] How it works: The server receives an HTTP POST request and temporarily saves the photos and videos to the specified directory.

[0970] Input: The photo or video file included in the HTTP POST request.

[0971] Output: Temporarily saved photo or video files.

[0972] Step 4:

[0973] The server analyzes the photos and videos.

[0974] How it works: The server's image analysis module uses OpenCV and TensorFlow to analyze photos and videos, recognizing scenery and people, and dividing videos into frames to extract movement.

[0975] Input: A saved photo or video file.

[0976] Output: Recognized scenery, people, and movement feature information.

[0977] Step 5:

[0978] The server generation AI generates a 3D metaverse space.

[0979] How it works: The extracted feature information is input into a generative AI, which then uses GAN and 3D reconstruction algorithms to generate a 3D metaverse space.

[0980] Input: Extracted feature information.

[0981] Output: The generated 3D metaverse spatial data.

[0982] Step 6:

[0983] Stores metaverse spatial data in a database.

[0984] How it works: The generated 3D Metaverse spatial data is stored in a database, and an identifier is generated and linked to the user's profile.

[0985] Input: Generated 3D metaverse spatial data.

[0986] Output: 3D metaverse spatial data and identifiers stored in a database.

[0987] Step 7:

[0988] A user accesses the metaverse space.

[0989] How it works: A user puts on a VR headset or AR-enabled device and logs into the server through a dedicated app. A list of saved Metaverse spaces is displayed, and the user can select a specific experience. The selected space data is streamed in real time.

[0990] Input: Login information to the server, identifier of the selected metaverse space.

[0991] Output: Metaverse spatial footage streamed in real time to the user's device.

[0992] Step 8:

[0993] The user's emotions are captured and the server tailors the experience.

[0994] How it works: The user's device captures facial expressions and voice and sends them to the emotion engine. The emotion engine analyzes the emotional data and sends it to the server. The server uses this data to adjust the visual and audio effects in the metaverse space in real time.

[0995] Input: Captured facial and voice data, analyzed emotion data.

[0996] Output: A metaverse spatial experience with visual and audio effects tuned in real time.

[0997] (Application example 2)

[0998] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0999] In modern society, taking photos and videos has become an important means of preserving and re-experiencing memories. However, it is difficult to fully reproduce the emotions and atmosphere of a moment with two-dimensional photos and videos, and they are unable to fully recreate the excitement and joy felt by the user. It is also difficult for a user to relive an experience captured at a specific location with a sense of presence from another location. Furthermore, no system has been available to date that can provide a customized experience based on the user's emotions. There is a need for a system that can solve these issues and provide a more immersive and emotionally responsive real-time experience.

[1000] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1001] In this invention, the server includes means for receiving photos or videos acquired from a user device and analyzing scenery, people, and movements from the photos or videos, means for a generation AI to generate a 3D metaverse space based on the analysis results, means for saving the generated 3D metaverse space and providing it for user access, means for acquiring emotional data from the user device, and means for adjusting visual and audio effects in the metaverse space in real time based on the acquired emotional data. This allows users to experience a highly immersive metaverse space based on the photos and videos they have taken, and allows them to enjoy a real-time experience customized to their emotions at the time.

[1002] A "user terminal" is a device that a user uses to take photos and videos and send the data to a server, and includes smartphones, tablets, digital cameras, etc.

[1003] "Server" refers to a computer system that receives and analyzes data sent from user terminals, and generates, stores, and provides the metaverse space.

[1004] "Generative AI" is a technology that includes deep learning models and 3D reconstruction algorithms for generating a three-dimensional metaverse space based on data received from user devices.

[1005] "Emotion data" is information about the emotional state obtained by analyzing the user's facial expressions and voice.

[1006] "Means for real-time adjustment" refers to technology that instantly adjusts visual and sound effects within the metaverse space based on acquired emotional data.

[1007] A "metaverse space" is a three-dimensional virtual space where users can experience virtual reality.

[1008] "Visual and audio effects" refers to visual and audio effects that enhance the user experience, such as adjusting lighting or changing background music.

[1009] This invention is a system that recreates the scenery, people, and movements of a moment in a three-dimensional metaverse space based on photos and videos taken by the user, and further adjusts the experience by recognizing the user's emotions. Detailed embodiments for carrying out this invention are described below.

[1010] First, a user terminal is a device that includes a smartphone, tablet, digital camera, etc., that allows the user to take photos and videos and send the data to a server. When a user takes a photo or video, the data is sent to the server through a dedicated application. At this stage, the user's facial expressions and voice are also captured and sent as emotion data.

[1011] The server receives photo, video, and emotion data sent from the user's device. It analyzes the received data and uses computer vision technology to extract scenery, people, and movements. Specifically, image analysis is performed using libraries such as OpenCV and TensorFlow. Based on the analysis results, a generative AI generates a 3D metaverse space. The generative AI includes deep learning models and 3D reconstruction algorithms.

[1012] The generated metaverse space is stored on a server and made accessible to users. Users can access the generated metaverse space and experience it with a high level of immersion using a virtual reality headset or an augmented reality-enabled device.

[1013] Furthermore, the emotion engine analyzes the user's facial expressions and voice to recognize their emotional state, and adjusts the visual and audio effects in the metaverse space in real time based on the recognized emotion. For example, if the user is emotional, the background music can be changed to a more uplifting one, or the lighting can be adjusted to be more vibrant.

[1014] Specific examples

[1015] Users take photos of new shoes and upload them to a server using a dedicated application. The server analyzes the received photos, generates a 3D model of the shoe, and stores it for users to access. Users can then put on a virtual reality headset and enter a virtual store to try on the generated 3D shoe model. The user's facial expressions and voice are analyzed by an emotion engine, which adjusts the lighting and background music in the store based on the emotion data.

[1016] Prompt Sentence Examples

[1017] "Generate a realistic 3D model based on a photograph of a new running shoe. The model should accurately reproduce the shoe's details and specific features (e.g., sole design, color tone)."

[1018] This allows users to experience a highly immersive metaverse space based on the photos and videos they take, and allows them to enjoy a real-time experience customized to their emotions at the time.

[1019] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1020] Step 1:

[1021] The user takes a photo or video.

[1022] Input: Photos and videos taken by users using devices such as smartphones and tablets.

[1023] Specific operation: The user launches the photo app and takes photos or videos while traveling or at an event. The captured data is saved in the device's local storage.

[1024] Output: Photo and video data saved in local storage.

[1025] Step 2:

[1026] The user uploads data.

[1027] Input: Photo and video data stored on the user's device.

[1028] How it works: The user opens the application and selects the photos and videos they have taken for uploading. The selected data is sent to the server as an HTTP POST request.

[1029] Output: Photo and video data sent to the server.

[1030] Step 3:

[1031] The server receives the data.

[1032] Input: HTTP POST request sent from the user's device.

[1033] Specific operation: The server receives the HTTP request and temporarily saves the sent photo or video files in the specified directory.

[1034] Output: Temporarily saved photo and video files.

[1035] Step 4:

[1036] The server analyzes the photos and videos.

[1037] Input: Temporarily saved photo and video files.

[1038] How it works: The server's image analysis module analyzes photo and video data using computer vision technologies such as OpenCV and TensorFlow, recognizing scenery, people, and movements, and extracting features from each frame.

[1039] Output: Analyzed feature information (scenery, people, movement) data.

[1040] Step 5:

[1041] The server uses generation AI to generate a three-dimensional metaverse space.

[1042] Input: Parsed feature information data.

[1043] Specific operation: The server uses generative AI, including deep learning models and 3D reconstruction algorithms, to generate a three-dimensional metaverse space based on the analysis results.

[1044] Output: Generated 3D metaverse spatial data.

[1045] Step 6:

[1046] The server stores and provides the generated three-dimensional metaverse space.

[1047] Input: Generated 3D Metaverse spatial data.

[1048] Specific operation: The server stores the generated 3D metaverse space data in a database, links the identifier to the user's profile, and provides the stored data for the user to access.

[1049] Output: Metaverse spatial data stored in a database.

[1050] Step 7:

[1051] Emotion data is acquired from the user's device.

[1052] Input: Facial expressions and voice captured by the user's device.

[1053] Specific operation: The user device captures facial expressions and voices during the user's experience and sends them to the emotion engine.

[1054] Output: Facial and speech data captured and sent to the emotion engine.

[1055] Step 8:

[1056] Adjusting the metaverse space in real time based on emotional data.

[1057] Input: Acquired emotion data.

[1058] Specific operation: The server analyzes the data sent from the emotion engine, recognizes the user's emotional state, and instantly adjusts the visual and sound effects in the metaverse space based on the recognized emotion.

[1059] Output: A real-time adjusted metaverse space.

[1060] Step 9:

[1061] A user accesses the metaverse space.

[1062] Input: User login information and metaverse space identifier.

[1063] Specific operation: The user puts on a head-mounted display or augmented reality device and launches a dedicated application. After logging in, the user selects a specific space from a list of saved metaverse spaces to begin the experience. The selected metaverse space data is then streamed in real time from the server and displayed to the user.

[1064] Output: A real-time display of the metaverse space experienced by the user.

[1065] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1066] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1067] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1068] [Third embodiment]

[1069] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1070] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1071] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1072] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1073] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1074] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1075] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1076] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1077] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1078] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1079] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1080] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1081] This invention is a system that recreates the scenery, people, and movements of a moment in a metaverse space based on photos and videos taken by the user. This system utilizes the user's device, a server, and generation AI to provide the user with an immersive experience.

[1082] System configuration

[1083] 1. User Device:

[1084] They act as devices that allow users to take photos and videos, including smartphones, tablets, and digital cameras.

[1085] A dedicated application is installed to allow users to send captured image data to a server.

[1086] Users access the saved metaverse space and use VR headsets or AR-enabled devices to experience it.

[1087] 2. Server:

[1088] It has a component that receives photo and video data sent from the user's device.

[1089] Computer vision techniques are used to analyze the received data and extract scenery, people, and movement.

[1090] Based on the extracted information, a 3D metaverse space is generated using generative AI.

[1091] The generated metaverse spatial data is stored in a database and provided for users to access.

[1092] 3. Generation AI:

[1093] Based on the analysis results, it includes deep learning models and 3D reconstruction algorithms to generate 3D models that reproduce real-world scenes, people, and movements.

[1094] System Operation

[1095] Data acquisition and transmission (user's device)

[1096] User takes a photo or video:

[1097] A user takes photos and videos using the device's camera app while traveling or at an event.

[1098] The captured data is stored in the device's local storage.

[1099] User uploads data:

[1100] The user opens the dedicated app and goes to the upload screen of the metaverse generation service.

[1101] Select the saved photo or video file and press the upload button.

[1102] The app sends the selected file to the server as an HTTP POST request.

[1103] Data reception and analysis (server)

[1104] Server receives data:

[1105] The server receives the HTTP request and retrieves the transmitted file data.

[1106] The received file is stored in a directory for temporary storage.

[1107] The server analyzes the photos and videos:

[1108] The server's image analysis module analyzes the transmitted photo and video data.

[1109] Image analysis uses computer vision libraries such as OpenCV and TensorFlow to recognize scenery and people.

[1110] In video analysis, a video is divided into frames, and each frame is subjected to image analysis to extract movement.

[1111] Generation and storage of 3D metaverse space (server)

[1112] The server generates a 3D metaverse space using generative AI:

[1113] The extracted feature information is input into a generation AI to generate a 3D metaverse space.

[1114] Generative AI uses GANs and 3D reconstruction algorithms to recreate realistic situations.

[1115] The server saves the generated metaverse space:

[1116] The generated 3D data is stored in a database along with metadata.

[1117] Generate an identifier (ID) for the stored spatial data and link it to the user's profile.

[1118] Providing a metaverse experience (user device)

[1119] User accesses the metaverse space:

[1120] The user puts on a VR headset or AR-compatible device and launches a dedicated app.

[1121] Log in to the server from the app and view a list of saved metaverse spaces.

[1122] Select a specific metaverse space from the list provided to begin the experience.

[1123] The app streams selected metaverse spatial data from the server and displays it to the user in real time.

[1124] Specific examples

[1125] 1. Relive your travel memories:

[1126] A user takes a photo or video in front of the Eiffel Tower while traveling in Paris (user's device).

[1127] The taken photo is uploaded by dragging and dropping it onto the upload screen of the metaverse generation service (user's device).

[1128] The server receives the photo data sent from the user's terminal and temporarily stores it (server).

[1129] The server's image analysis module recognizes the Eiffel Tower and the surrounding scenery from the sent photo and extracts its features (server).

[1130] The generative AI generates a 3D model of the Eiffel Tower based on the extracted features and recreates the background scenery (server).

[1131] The generated 3D model and scenery data are stored in a database and linked to the user's profile (server).

[1132] Users put on a VR headset at home and open a dedicated app to access the Eiffel Tower metaverse space (on their device).

[1133] Users can move freely within the metaverse and experience the Eiffel Tower and its surroundings in 360 degrees (on their device).

[1134] 2. Recreating child growth records:

[1135] A user records a video of their child's first birthday (on their device).

[1136] The captured video is uploaded to the metaverse generation service (user's device).

[1137] The server receives the video and analyzes the birthday cake and the child's movements (server).

[1138] Based on the analyzed information, the generation AI generates a 3D metaverse space for the birthday party (server).

[1139] The generated 3D data is stored in a database and made accessible to users (server).

[1140] The user uses a VR headset to relive their child's first birthday (on the user's device).

[1141] The system of the present invention allows users to realistically preserve and experience memories in a highly immersive alternative to two-dimensional photos and videos.

[1142] The processing flow will be explained below.

[1143] Step 1:

[1144] Data acquisition (user device)

[1145] Users use the camera app on their devices to take photos or videos of scenery at their travel destinations or events.

[1146] The captured data is stored in the device's local storage.

[1147] Step 2:

[1148] Data transmission (user's device)

[1149] Users open the dedicated app and select the photos and videos they have taken.

[1150] When you press the upload button, the app sends the selected data to the server as an HTTP POST request.

[1151] Step 3:

[1152] Data reception (server)

[1153] The server receives an HTTP POST request to retrieve the photo or video file.

[1154] To temporarily store the received data, save it in the specified directory.

[1155] Step 4:

[1156] Image and video analysis (server)

[1157] The server uses computer vision libraries (e.g., OpenCV and TensorFlow) to analyze the stored photo and video data.

[1158] In the case of images, the server extracts features such as scenery, people, and objects from the image.

[1159] In the case of video, the server divides the video into frames, analyzes each frame, and extracts movement.

[1160] Step 5:

[1161] 3D Metaverse Space Generation (Server)

[1162] Based on the analysis results, the server generates a 3D metaverse space using generative AI (e.g., GAN or 3D reconstruction algorithms).

[1163] Generative AI uses extracted features to recreate scenes, people, and movements.

[1164] Step 6:

[1165] Metaverse space storage (server)

[1166] The server stores the generated 3D metaverse spatial data in a database.

[1167] Generate an identifier (ID) for the saved space and link it to the user's profile.

[1168] Step 7:

[1169] Providing a metaverse experience (user device)

[1170] The user puts on a VR headset or AR-compatible device and launches a dedicated app.

[1171] Users log in to the server through the app and view a list of saved metaverse spaces.

[1172] Select a specific metaverse space from the list provided to begin the experience.

[1173] The app streams selected metaverse spatial data in real time from the server and displays it to the user.

[1174] In this way, users can experience an immersive three-dimensional metaverse space generated from two-dimensional photos and videos.

[1175] Example 1

[1176] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1177] Conventional recording methods using photographs and videos forced users to rely on two-dimensional information, making it difficult to recreate the atmosphere or three-dimensional experience of the scene. Furthermore, there were limited ways to preserve detailed memories of specific events or travel destinations and experience them later in real life. Furthermore, existing systems required complex processes for data analysis and reproduction, making practical application difficult.

[1178] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1179] In this invention, the server includes: means for receiving photos or videos acquired from a user terminal and analyzing scenery, people, and movements from the photos or videos; means for a generation AI to generate a 3D metaverse space based on the analysis results; means for saving the generated 3D metaverse space and providing it for user access; means for a user to take photos or videos and upload the data to the server using a dedicated application; means for the server to save the transmitted photos or videos in a temporary directory and analyze them using computer vision technology; and means for saving the generated metaverse space data in a database and linking the identifier to a user profile. This allows users to have a more immersive experience, enabling the realistic reproduction of on-site atmospheres and three-dimensional experiences.

[1180] "User terminal" refers to any digital device that allows a user to take and save photos and videos, and specific examples include smartphones and digital cameras.

[1181] A "server" refers to a computer system that receives, analyzes, and stores data sent from a user terminal, and provides various services to the user.

[1182] "Photos or videos" refers to still images or video data taken by a user using a digital device.

[1183] "Analysis" refers to the process of recognizing scenery, people, and movements from photos or videos and extracting their respective characteristics.

[1184] "Scenery" refers to the natural or man-made scenery or backgrounds contained in photographs or videos.

[1185] "People" refers to human figures appearing in photographs and videos.

[1186] "Motion" refers to the movement and activity of people and objects in a photo or video.

[1187] "Generative AI" refers to artificial intelligence techniques used to generate 3D Metaverse spaces based on analyzed data, including GANs and 3D reconstruction algorithms.

[1188] "3D Metaverse space" refers to a digital space in which real-world scenery, people, and movements are reproduced in three dimensions within a virtual space.

[1189] "Dedicated application" refers to software that users use to upload photos and videos to a server.

[1190] The "temporary directory" refers to a folder where the server temporarily stores data received.

[1191] "Computer vision technology" refers to the technology that allows computers to analyze images and videos and extract information. Examples include OpenCV and TensorFlow.

[1192] "Database" refers to a system for efficiently storing and managing the generated metaverse spatial data. A specific example is MongoDB.

[1193] "Identifier" refers to an ID that uniquely identifies stored metaverse spatial data.

[1194] "User profile" refers to a digital data structure for storing information and data related to a user.

[1195] "VR headset" refers to a device used by a user to experience a virtual reality space, including, for example, Oculus Rift.

[1196] "AR-enabled device" means a device used to experience an augmented reality space, including, for example, Microsoft HoloLens.

[1197] MODE FOR CARRYING OUT THE INVENTION

[1198] This invention is a system that recreates the scenery, people, and movements of a moment in a metaverse space based on photos and videos taken by the user. This system utilizes the user's device, a server, and generation AI to provide the user with an immersive experience.

[1199] 1. System Configuration

[1200] User terminal

[1201] A user device is a general term for a digital device that allows a user to take and save photos and videos, including smartphones and digital cameras. A dedicated application (e.g., the Metaverse Creator app) is installed on the device to transmit the captured data to a server. Users can access and experience the saved Metaverse space using a VR headset (e.g., Oculus Rift) or an AR-enabled device (e.g., Microsoft HoloLens).

[1202] server

[1203] The server receives photo and video data sent from user devices, analyzes it, and generates the metaverse space. The server includes the following main components:

[1204] Data receiving module: Receives HTTP requests and retrieves the transmitted file data.

[1205] Temporary storage: The directory where received data is temporarily stored (e.g. / tmp / uploads).

[1206] Image analysis module: Uses computer vision libraries such as OpenCV and TensorFlow to extract scenery, people, and movement from photo and video data.

[1207] Generative AI module: Uses GAN and 3D reconstruction algorithms to generate a 3D metaverse space based on extracted data.

[1208] Database: A database (e.g., MongoDB) to store the generated metaverse spatial data and associated metadata.

[1209] 2. System Operation

[1210] Data acquisition and transmission by the user's device

[1211] A user takes photos and videos using a smartphone or digital camera. For example, while traveling, the user takes a photo of the Eiffel Tower using the iPhone camera app and saves it to the camera roll. The user then launches the MetaverseCreator app, selects the photo, and uploads it. The app then sends the photo data to the server as an HTTP POST request.

[1212] Data reception and analysis by the server

[1213] The server receives the HTTP request and saves the data in a temporary directory. The image analysis module then analyzes the photo data, recognizing scenery such as the Eiffel Tower, people, and movements, and extracting feature points. This feature point information is stored as JSON data.

[1214] Creation and storage of 3D metaverse space

[1215] The server inputs the extracted feature information into a generative AI module, which then uses a GAN model and 3D reconstruction algorithm to generate a 3D structure of the Eiffel Tower and recreate the background scenery. The generated metaverse spatial data is stored in a database, and an identifier is generated and linked to the user's profile.

[1216] User experience of the metaverse space

[1217] Users put on a VR headset or AR-enabled device and launch the MetaverseViewer app. They log in within the app and view a list of Metaverse spaces linked to their profile. They select a specific Metaverse space and begin the experience. The app streams the Metaverse space data from the server and displays it on the user's device in real time.

[1218] Examples and prompts

[1219] For example, if a user uploads a photo of the Eiffel Tower taken during a trip to Paris, and a generative AI generates a 3D metaverse of the Eiffel Tower based on that photo, the actual prompt would look like this:

[1220] "Generate a 3D model of the Eiffel Tower based on the following photo."

[1221] "Reconstruct a scene from a child's birthday party using frames from this video."

[1222] The system of the present invention allows users to realistically preserve and experience memories in a highly immersive alternative to two-dimensional photos and videos.

[1223] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1224] Step 1:

[1225] A user takes a photo or video. A user takes a photo or video using a smartphone or digital camera. For example, while traveling, a user takes a photo of the Eiffel Tower using the camera app on their iPhone and saves it to their camera roll. The input is the camera app on the digital device, and the output is the photo or video data saved in local storage.

[1226] Step 2:

[1227] The user uploads data to the server using a dedicated application. The user launches the MetaverseCreator app, selects the photos and video files they have taken, and uploads them. The input is the photo or video file saved in local storage, and the output is the file data sent as an HTTP POST request. During this process, an upload progress bar is displayed.

[1228] Step 3:

[1229] The server receives the data. The server receives the HTTP request and saves the submitted file data and metadata in a temporary directory (e.g., / tmp / uploads). The input is the photo or video data included in the HTTP request body, and the output is the file saved in the temporary directory.

[1230] Step 4:

[1231] The server analyzes photos and videos. The server's image analysis module (e.g., OpenCV, TensorFlow) analyzes the photo and video data stored in a temporary directory. The input is the file stored in the temporary directory, and the output is feature information about the scenery, people, and actions stored in JSON format. Specifically, feature points of the Eiffel Tower and people are extracted from photos, and each frame of the video is analyzed to detect movement.

[1232] Step 5:

[1233] The server uses generative AI to generate a 3D metaverse space. The server inputs the extracted feature point information into a generative AI module (e.g., GAN or 3D reconstruction algorithm) to generate the 3D metaverse space. The input is JSON data of the feature point information, and the output is data in the generated 3D metaverse space. For example, a GAN model generates the 3D structure of the Eiffel Tower and recreates the background scenery.

[1234] Step 6:

[1235] The server saves the generated metaverse space in a database. The server saves the generated metaverse space data in a database (e.g., MongoDB) and links the identifier to the user profile. The input is the 3D metaverse space data and the user ID, and the output is the data saved in the database and the generated identifier.

[1236] Step 7:

[1237] A user accesses the Metaverse space by putting on a VR headset (e.g., Oculus Rift) or an AR-enabled device (e.g., Microsoft HoloLens) and launching the MetaverseViewer app. The input is the application on the device, and the output is the Metaverse space data streamed from the server. When the user selects a Metaverse space in the app and presses the "Start Experience" button, the app displays the data in real time, and the user enjoys an immersive experience.

[1238] (Application example 1)

[1239] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1240] There is a need for a method that allows users to upload photos and videos they have taken to the metaverse space through a dedicated application, and then share the 3D metaverse space generated from it with other users, thereby providing a visually immersive experience. However, existing technologies do not easily enable the generation and sharing of such advanced metaverse spaces, so a new solution is needed.

[1241] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1242] In this invention, the server includes means for receiving photos or videos acquired from a user terminal and analyzing scenery, people, and movements from the photos or videos, means for a generation AI to generate a 3D metaverse space based on the analysis results, means for saving the generated 3D metaverse space and providing it for users to access, means for users to upload and share photos or videos using a smartphone application, and means for sharing the generated 3D metaverse space with friends and family. This allows users to generate realistic 3D metaverse spaces based on photos and videos they have taken and easily share them with other users, providing a visually realistic and immersive experience.

[1243] "User terminal" refers to a device that takes photos and videos and has installed an application for sending that data to a server.

[1244] A "server" refers to a computer system that receives data sent from a user terminal, analyzes it, and performs the necessary processing.

[1245] "Generative AI" refers to artificial intelligence technology for generating a three-dimensional metaverse space based on the analysis of photos and videos sent from user devices.

[1246] "3D Metaverse space" refers to a three-dimensional virtual space generated based on real-world scenery, people, and movements.

[1247] "Smartphone application" refers to software that is installed on a smartphone and allows users to upload photos and videos, share and experience the metaverse space.

[1248] "Analyzing" refers to the process of extracting specific information (scenery, people, movements, etc.) from photos and videos.

[1249] "Saving" refers to the act of storing the generated 3D metaverse spatial data in a state that allows it to be accessed later.

[1250] "Providing" refers to the process of allowing a user to access a saved three-dimensional metaverse space.

[1251] "Sharing" refers to the act of exchanging the generated 3D metaverse space with other users in an easily accessible manner.

[1252] "Friends and Family" refers to close people with whom the user wants to share.

[1253] This invention is a system that recreates the scenery, people, and movements of a moment in a 3D metaverse space based on photos and videos taken by the user. This system utilizes the user's device, a server, and a generation AI to provide the user with an immersive experience.

[1254] System configuration

[1255] It consists of the following main components:

[1256] 1. User Device:

[1257] Camera device: A device that allows a user to take photos or videos. Examples include smartphones, tablets, and digital cameras.

[1258] Dedicated application: An application is installed to send the captured data to the server. VR headsets and AR-compatible devices are also used to access and experience the generated 3D metaverse space.

[1259] 2. Server:

[1260] Receiving module: Contains a component that receives and temporarily stores photo and video data sent from the user device.

[1261] Image analysis module: Analyzes received photo and video data using computer vision technology, specifically using libraries such as OpenCV and TensorFlow.

[1262] Generative AI module: Generates a 3D metaverse space based on the extracted information. This uses deep learning models, including GANs and 3D reconstruction algorithms.

[1263] Database: Stores the generated 3D metaverse spatial data and makes it accessible to users.

[1264] 3. Generation AI:

[1265] Deep learning model: A technology for generating 3D models that reproduce real-world scenery, people, and movements based on analysis results sent from the user's device.

[1266] 3D reconstruction algorithm: Creates a realistic 3D space based on the analysis results.

[1267] System Operation

[1268] 1. Data acquisition and transmission (user terminal):

[1269] Users take photos and videos using their smartphones and upload the data to the server via a dedicated application. Uploads are made via HTTP POST requests.

[1270] 2. Data reception and analysis (server):

[1271] The server receives and temporarily stores the photos and video data sent from the user's device. It then uses computer vision technology to analyze the scenery, people, and movements from the received data. Libraries such as OpenCV and TensorFlow are used for this.

[1272] 3. Generation and storage of 3D metaverse space (server):

[1273] The analysis results are input into the generation AI to generate a 3D metaverse space. The generated 3D data is stored in a database and made available for users to access.

[1274] 4. Providing a Metaverse experience (user device):

[1275] Users can access the generated 3D metaverse space through a dedicated application using a VR headset or AR-compatible device, and can also share the generated metaverse space with friends and family.

[1276] Specific examples

[1277] Travel Reenactment:

[1278] Users upload photos taken in front of the Eiffel Tower during their travels. The server analyzes the photos and uses generative AI to generate a 3D metaverse of the Eiffel Tower and its surroundings. Users can access this metaverse by wearing a VR headset and enjoy a 360-degree experience.

[1279] Recreating a child's growth record:

[1280] Users upload videos of their child's first birthday. The server analyzes the videos and uses generative AI to generate a 3D space of the birthday party. Users can access this recreated space and relive their child's growth and memories.

[1281] Prompt Sentence Examples

[1282] "Upload a photo of the Eiffel Tower and generate a realistic 3D metaverse space based on this photo."

[1283] This system allows users to easily recreate their own photos and videos in a 3D metaverse space and share that space with others, providing a visually immersive experience.

[1284] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1285] Step 1:

[1286] The user takes a photo or video

[1287] Users take photos and videos using the camera app on their smartphones.

[1288] Input: Photo or video data from a smartphone camera app

[1289] Output: Photo or video files stored in the smartphone's local storage

[1290] Step 2:

[1291] User uploads data

[1292] The user opens the application, selects the photos and video files they have taken, and presses the upload button, which sends these files to the server as an HTTP POST request.

[1293] Input: Photo or video files selected through a dedicated application, server upload endpoint

[1294] Output: HTTP POST request sent to the server

[1295] Step 3:

[1296] The server receives the data

[1297] The server receives the HTTP POST request, retrieves the sent file data, and saves it in a temporary directory.

[1298] Input: Photo or video data received via an HTTP POST request

[1299] Output: File data saved in temporary directory

[1300] Step 4:

[1301] The server analyzes the photos and videos.

[1302] The server's image analysis module uses OpenCV and TensorFlow to analyze the received photo and video data. Image analysis recognizes scenery and people, while video analysis divides the video into frames to extract movement.

[1303] Input: Photo or video data saved in the temporary directory

[1304] Output: Scenery, people, and movement information as analysis results

[1305] Step 5:

[1306] The server generates a 3D metaverse space using generation AI.

[1307] The server inputs the analysis results into the generation AI, which then generates a 3D metaverse space using GAN and 3D reconstruction algorithms.

[1308] Input: Scenery, people, and movement information as analysis results

[1309] Output: Generated 3D metaverse spatial data

[1310] Step 6:

[1311] The server stores the generated 3D metaverse spatial data.

[1312] The server stores the generated 3D metaverse spatial data in a database and makes it accessible to users.

[1313] Input: Generated 3D metaverse spatial data, user identification information

[1314] Output: 3D metaverse spatial data stored in a database, metaverse space linked to user profile

[1315] Step 7:

[1316] A user accesses the metaverse space

[1317] Users use a VR headset or AR-compatible device to log in through a dedicated application, select one from a list of saved 3D metaverse spaces, and begin the experience.

[1318] Input: User login information, list of 3D metaverse spaces stored in the database

[1319] Output: 3D metaverse space streamed to user's device

[1320] Step 8:

[1321] Users share the metaverse space

[1322] Users can share the 3D metaverse space they create with friends and family through a dedicated application, generating a shared link and granting access rights to specific users.

[1323] Input: 3D metaverse space linked to user profile, shared identity information

[1324] Output: Shared link and access rights to the shared 3D metaverse space

[1325] The above are the specific processing steps in the system for implementing this invention. At each step, various data processing and calculations are performed based on the input data, and the results are passed on to the next step, thereby operating the entire system.

[1326] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1327] The present invention is a system that recreates the scenery, people, and movements of a user in a metaverse space based on photos and videos taken by the user, and further adjusts the experience by recognizing the user's emotions. The following describes the detailed operation of the system.

[1328] System configuration

[1329] 1. User Device:

[1330] They act as devices that allow users to take photos and videos, including smartphones, tablets, and digital cameras.

[1331] A dedicated application is installed to allow users to send captured image data to a server.

[1332] It also has the function of capturing the user's facial expressions and voice and sending them to the emotion engine.

[1333] Users access the saved metaverse space and use VR headsets or AR-enabled devices to experience it.

[1334] 2. Server:

[1335] It has a component that receives photo and video data sent from the user's device, as well as emotion data recognized by the emotion engine.

[1336] Computer vision techniques are used to analyze the received data and extract scenery, people, and movement.

[1337] Based on the extracted information, a 3D metaverse space is generated using generative AI.

[1338] The generated metaverse spatial data is stored in a database and provided for users to access.

[1339] Tailor your metaverse experience in real time based on data from the emotion engine.

[1340] 3. Generation AI:

[1341] Based on the analysis results, it includes deep learning models and 3D reconstruction algorithms to generate 3D models that reproduce real-world scenes, people, and movements.

[1342] 4. Emotion Engine:

[1343] Recognizes emotions by analyzing the user's facial expressions and voice.

[1344] Adjust visual and audio effects within the metaverse space based on the emotions recognized.

[1345] System Operation

[1346] Data acquisition and transmission (user's device)

[1347] User takes a photo or video:

[1348] A user takes photos and videos using the device's camera app while traveling or at an event.

[1349] The captured data is stored in the device's local storage.

[1350] User uploads data:

[1351] The user opens the dedicated app and selects the photos and videos they have taken.

[1352] When you press the upload button, the app sends the selected data to the server as an HTTP POST request.

[1353] Data reception and analysis (server)

[1354] Server receives data:

[1355] The server receives an HTTP POST request to retrieve the photo or video file.

[1356] To temporarily store the received data, save it in the specified directory.

[1357] The server analyzes the photos and videos:

[1358] The server's image analysis module analyzes the transmitted photo and video data.

[1359] Image analysis uses computer vision libraries such as OpenCV and TensorFlow to recognize scenery and people.

[1360] In video analysis, a video is divided into frames, and each frame is subjected to image analysis to extract movement.

[1361] 3D Metaverse Space Generation (Server)

[1362] The server generates a 3D metaverse space using generative AI:

[1363] The extracted feature information is input into a generation AI to generate a 3D metaverse space.

[1364] Generative AI uses GANs and 3D reconstruction algorithms to recreate realistic situations.

[1365] Metaverse space storage (server)

[1366] The server stores the generated 3D metaverse space:

[1367] The server stores the generated 3D metaverse spatial data in a database.

[1368] Generate an identifier (ID) for the stored spatial data and link it to the user's profile.

[1369] Emotion recognition and user experience adjustment (emotion engine and server)

[1370] Emotion engine recognizes emotions:

[1371] The user device captures the user's facial expressions and voice and sends them to the emotion engine.

[1372] The emotion engine analyzes this data and recognizes the user's emotional state.

[1373] Emotion-based experience tailoring:

[1374] The emotion data obtained from the emotion engine is received by the server.

[1375] The server changes the visual and sound effects in the metaverse space in real time based on the emotional data.

[1376] For example, if the user is impressed, the scenery will be made more vivid and the background music will be changed to something more exciting.

[1377] Providing a metaverse experience (user device)

[1378] User accesses the metaverse space:

[1379] The user puts on a VR headset or AR-compatible device and launches a dedicated app.

[1380] Users log in to the server through the app and view a list of saved metaverse spaces.

[1381] Select a specific metaverse space from the list provided to begin the experience.

[1382] The app streams selected metaverse spatial data in real time from the server and displays it to the user.

[1383] Specific examples

[1384] 1. Travel memory recall and emotional regulation:

[1385] A user takes a photo or video in front of the Eiffel Tower while traveling in Paris (user's device).

[1386] The taken photo is uploaded by dragging and dropping it onto the upload screen of the metaverse generation service (user's device).

[1387] The server receives the photo data sent from the user's device and temporarily stores it (server).

[1388] The server's image analysis module recognizes the Eiffel Tower and the surrounding scenery from the sent photo and extracts its features (server).

[1389] The generative AI generates a 3D model of the Eiffel Tower based on the extracted features and recreates the background scenery (server).

[1390] The generated 3D model and scenery data are stored in a database and linked to the user's profile (server).

[1391] Users put on a VR headset at home and open a dedicated app to access the Eiffel Tower metaverse space (on their device).

[1392] While the user is experiencing the experience, the emotion engine analyzes the user's facial expressions and if it recognizes emotion, the server adjusts the effects and changes the background music to an emotional one (server).

[1393] 2. Recapitulation of child development records and emotional regulation:

[1394] A user records a video of their child's first birthday (on their device).

[1395] The captured video is uploaded to the metaverse generation service (user's device).

[1396] The server receives the video and analyzes the birthday cake and the child's movements (server).

[1397] Based on the analyzed information, the generation AI generates a 3D metaverse space for the birthday party (server).

[1398] The generated 3D data is stored in a database and made accessible to users (server).

[1399] The user uses a VR headset to relive their child's first birthday (on the user's device).

[1400] If the emotion engine recognizes the user's joy, the server will brighten the lights in the space and make the background music more enjoyable (server).

[1401] The system of the present invention allows users to realistically preserve memories in a highly immersive alternative to two-dimensional photos and videos, and allows for customized experiences that reflect their emotional state at any given time.

[1402] The processing flow will be explained below.

[1403] Step 1:

[1404] Data acquisition (user device)

[1405] Users use the camera app on their devices to take photos or videos of scenery at their travel destinations or events.

[1406] The captured data is stored in the device's local storage.

[1407] Step 2:

[1408] Data transmission (user's device)

[1409] Users open the dedicated app and select the photos and videos they have taken.

[1410] When you press the upload button, the app sends the selected data to the server as an HTTP POST request.

[1411] Step 3:

[1412] Data reception (server)

[1413] The server receives an HTTP POST request to retrieve the photo or video file.

[1414] To temporarily store the received data, save it in the specified directory.

[1415] Step 4:

[1416] Image and video analysis (server)

[1417] The server uses computer vision libraries (e.g., OpenCV and TensorFlow) to analyze the stored photo and video data.

[1418] In the case of images, the server extracts features such as scenery, people, and objects from the image.

[1419] In the case of video, the server divides the video into frames, analyzes each frame, and extracts movement.

[1420] Step 5:

[1421] 3D Metaverse Space Generation (Server)

[1422] Based on the analysis results, the server generates a 3D metaverse space using generative AI (e.g., GAN or 3D reconstruction algorithms).

[1423] Generative AI uses extracted features to recreate scenes, people, and movements.

[1424] Step 6:

[1425] Metaverse space storage (server)

[1426] The server stores the generated 3D metaverse spatial data in a database.

[1427] Generate an identifier (ID) for the saved space and link it to the user's profile.

[1428] Step 7:

[1429] Data acquisition for emotion recognition (user device)

[1430] The user puts on a VR headset or AR-compatible device and launches the dedicated app.

[1431] Sensors and microphones begin to operate to capture the user's facial expressions and voice.

[1432] Step 8:

[1433] Sending emotion data (user device)

[1434] The user's facial expression and voice data are sent to the server in real time.

[1435] This data is then forwarded to a dedicated endpoint to be analyzed by the emotion engine.

[1436] Step 9:

[1437] Receiving and analyzing emotion data (server and emotion engine)

[1438] The server passes the received facial expression and voice data to the emotion engine.

[1439] The emotion engine analyzes this data and recognizes the user's emotional state.

[1440] Step 10:

[1441] Emotion-based experience adjustment (server)

[1442] The server adjusts the visual and sound effects in the metaverse space in real time based on the emotional state data obtained from the emotion engine.

[1443] For example, if the user is moved, the scenery is made more vivid and the background music is changed to something more moving.

[1444] Step 11:

[1445] Providing a metaverse experience (user device)

[1446] Users log in to the server through a dedicated app and view a list of saved metaverse spaces.

[1447] Select a specific metaverse space from the list provided to begin the experience.

[1448] The app streams selected metaverse spatial data in real time from the server and displays it to the user.

[1449] In this way, users can experience an immersive three-dimensional metaverse space generated from two-dimensional photos and videos, and further customize it to suit their emotions at the time.

[1450] Example 2

[1451] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1452] Conventional methods simply record photos and videos, but have the problem of not being able to reflect or reproduce the user's emotions and experiences in real time. Furthermore, simple 2D data lacks a sense of immersion, making it difficult to reproduce the emotions and joy that users actually felt at the time.

[1453] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1454] In this invention, the server includes means for receiving photos or videos acquired from a user terminal and analyzing scenery, people, and movements from the photos or videos, means for a generation AI to generate a 3D metaverse space based on the analysis results, means for saving the generated 3D metaverse space and providing it for user access, means for capturing the user's facial expressions and voice and transmitting them to an emotion engine that recognizes emotions, and means for adjusting visual and audio effects in the metaverse space in real time based on the recognized emotion data. This allows users to realistically recreate past experiences with a high level of immersion and have a customized experience that matches their emotions at the time.

[1455] "User terminal" refers to a device that a user uses to take photos and videos and upload the data to a server, and includes smartphones, tablets, digital cameras, etc.

[1456] "Server" refers to a computer system that receives data sent from a user terminal, analyzes, processes, stores, and provides the data.

[1457] "Photos or videos" refers to still images or video files taken by a user using a device.

[1458] "Analysis of scenery, people, and movements" refers to the process of using computer vision technology to recognize scenery, people, and movements in photographs or video data and extract their characteristics.

[1459] "Generative AI" refers to deep learning models and 3D reconstruction algorithms for generating a 3D metaverse space based on extracted feature information.

[1460] "3D Metaverse space" refers to a three-dimensional virtual space generated by a generation AI to recreate real-world situations as virtual environments.

[1461] An "emotion engine" refers to software or hardware that analyzes a user's facial expressions and voice and recognizes the user's emotional state.

[1462] "Real-time adjustment of visual and audio effects" refers to the process of dynamically changing the images and audio displayed within the metaverse space to match the user's emotions based on recognized user emotional data.

[1463] "VR headset" refers to a device worn on the head to immerse a user in a virtual reality environment.

[1464] "AR-compatible device" refers to a device that uses augmented reality technology to overlay virtual objects on the real world.

[1465] "Computer vision technology" refers to technology that enables computers to analyze and understand the content of images and videos, and includes libraries such as OpenCV and TensorFlow.

[1466] This invention is a system that recreates the scenery, people, and movements of a moment in a 3D metaverse space based on photos and videos taken on a user's device, and adjusts the experience by recognizing the user's emotions. The detailed operation of the system is explained below.

[1467] A user takes a photo or video using a user device such as a smartphone, tablet, or digital camera. This data is temporarily stored on the user's device. The user then uses a dedicated application to select the photos or videos they have taken and upload them to the server. Uploads are made as HTTP POST requests.

[1468] The server temporarily stores the received photo or video data in a specified directory. The server's image analysis module then analyzes the data to recognize and extract scenery, people, and movements. This analysis uses computer vision technologies such as OpenCV and TensorFlow.

[1469] Based on the analysis results, the server uses generative AI to generate a 3D metaverse space. This generative AI uses GAN (generative adversarial networks) and 3D reconstruction algorithms to create a realistic virtual space. The generated 3D metaverse space data is then stored in a database, and an identifier is generated and linked to the user's profile.

[1470] Users put on a VR headset or AR-compatible device and log in to the server through a dedicated application. A list of saved 3D metaverse spaces is displayed, and the user selects a specific space to begin the experience. The selected space data is streamed in real time by the server and displayed on the user's device.

[1471] During the experience, the user's device captures facial expressions and voice and sends this data to the emotion engine. The emotion engine analyzes the user's emotional state and sends the recognized emotional data to the server. The server uses this data to adjust the visual and audio effects in the metaverse space in real time. For example, if the user is emotional, the scenery will be made more vivid and the background music will be more exciting.

[1472] For example, a user takes a photo in front of the Eiffel Tower while traveling in Paris and uploads the photo data to a server using a dedicated app. Based on this, the server generates a 3D metaverse space, which the user can access. The emotion engine recognizes the user's emotions during the experience, and the server adjusts the background music accordingly.

[1473] An example of a prompt sentence is as follows:

[1474] "Generate and display a 3D metaverse of the Eiffel Tower based on photos from a trip to Paris."

[1475] In this way, the system of the present invention allows users to realistically preserve memories in a highly immersive alternative to two-dimensional photos and videos, and allows for customized experiences based on emotional state.

[1476] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1477] Step 1:

[1478] The user takes a photo or video.

[1479] Action: The user takes a photo using a smartphone or digital camera.

[1480] Input: Real-world scenes and people.

[1481] Output: Captured photo or video files (formats such as JPEG or MP4).

[1482] Step 2:

[1483] The user uploads the captured data to the server.

[1484] How it works: Use the dedicated app to select the shooting data and press the upload button.

[1485] Input: The captured photo or video file.

[1486] Output: Sent as an HTTP POST request to the server.

[1487] Step 3:

[1488] The server receives and stores the data.

[1489] How it works: The server receives an HTTP POST request and temporarily saves the photos and videos to the specified directory.

[1490] Input: The photo or video file included in the HTTP POST request.

[1491] Output: Temporarily saved photo or video files.

[1492] Step 4:

[1493] The server analyzes the photos and videos.

[1494] How it works: The server's image analysis module uses OpenCV and TensorFlow to analyze photos and videos, recognizing scenery and people, and dividing videos into frames to extract movement.

[1495] Input: A saved photo or video file.

[1496] Output: Recognized scenery, people, and movement feature information.

[1497] Step 5:

[1498] The server generation AI generates a 3D metaverse space.

[1499] How it works: The extracted feature information is input into a generative AI, which then uses GAN and 3D reconstruction algorithms to generate a 3D metaverse space.

[1500] Input: Extracted feature information.

[1501] Output: The generated 3D metaverse spatial data.

[1502] Step 6:

[1503] Stores metaverse spatial data in a database.

[1504] How it works: The generated 3D Metaverse spatial data is stored in a database, and an identifier is generated and linked to the user's profile.

[1505] Input: Generated 3D metaverse spatial data.

[1506] Output: 3D metaverse spatial data and identifiers stored in a database.

[1507] Step 7:

[1508] A user accesses the metaverse space.

[1509] How it works: A user puts on a VR headset or AR-enabled device and logs into the server through a dedicated app. A list of saved Metaverse spaces is displayed, and the user can select a specific experience. The selected space data is streamed in real time.

[1510] Input: Login information to the server, identifier of the selected metaverse space.

[1511] Output: Metaverse spatial footage streamed in real time to the user's device.

[1512] Step 8:

[1513] The user's emotions are captured and the server tailors the experience.

[1514] How it works: The user's device captures facial expressions and voice and sends them to the emotion engine. The emotion engine analyzes the emotional data and sends it to the server. The server uses this data to adjust the visual and audio effects in the metaverse space in real time.

[1515] Input: Captured facial and voice data, analyzed emotion data.

[1516] Output: A metaverse spatial experience with visual and audio effects tuned in real time.

[1517] (Application example 2)

[1518] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1519] In modern society, taking photos and videos has become an important means of preserving and re-experiencing memories. However, it is difficult to fully reproduce the emotions and atmosphere of a moment with two-dimensional photos and videos, and they are unable to fully recreate the excitement and joy felt by the user. It is also difficult for a user to relive an experience captured at a specific location with a sense of presence from another location. Furthermore, no system has been available to date that can provide a customized experience based on the user's emotions. There is a need for a system that can solve these issues and provide a more immersive and emotionally responsive real-time experience.

[1520] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1521] In this invention, the server includes means for receiving photos or videos acquired from a user device and analyzing scenery, people, and movements from the photos or videos, means for a generation AI to generate a 3D metaverse space based on the analysis results, means for saving the generated 3D metaverse space and providing it for user access, means for acquiring emotional data from the user device, and means for adjusting visual and audio effects in the metaverse space in real time based on the acquired emotional data. This allows users to experience a highly immersive metaverse space based on the photos and videos they have taken, and allows them to enjoy a real-time experience customized to their emotions at the time.

[1522] A "user terminal" is a device that a user uses to take photos and videos and send the data to a server, and includes smartphones, tablets, digital cameras, etc.

[1523] "Server" refers to a computer system that receives and analyzes data sent from user terminals, and generates, stores, and provides the metaverse space.

[1524] "Generative AI" is a technology that includes deep learning models and 3D reconstruction algorithms for generating a three-dimensional metaverse space based on data received from user devices.

[1525] "Emotion data" is information about the emotional state obtained by analyzing the user's facial expressions and voice.

[1526] "Means for real-time adjustment" refers to technology that instantly adjusts visual and sound effects within the metaverse space based on acquired emotional data.

[1527] A "metaverse space" is a three-dimensional virtual space where users can experience virtual reality.

[1528] "Visual and audio effects" refers to visual and audio effects that enhance the user experience, such as adjusting lighting or changing background music.

[1529] This invention is a system that recreates the scenery, people, and movements of a moment in a three-dimensional metaverse space based on photos and videos taken by the user, and further adjusts the experience by recognizing the user's emotions. Detailed embodiments for carrying out this invention are described below.

[1530] First, a user terminal is a device that includes a smartphone, tablet, digital camera, etc., that allows the user to take photos and videos and send the data to a server. When a user takes a photo or video, the data is sent to the server through a dedicated application. At this stage, the user's facial expressions and voice are also captured and sent as emotion data.

[1531] The server receives photo, video, and emotion data sent from the user's device. It analyzes the received data and uses computer vision technology to extract scenery, people, and movements. Specifically, image analysis is performed using libraries such as OpenCV and TensorFlow. Based on the analysis results, a generative AI generates a 3D metaverse space. The generative AI includes deep learning models and 3D reconstruction algorithms.

[1532] The generated metaverse space is stored on a server and made accessible to users. Users can access the generated metaverse space and experience it with a high level of immersion using a virtual reality headset or an augmented reality-enabled device.

[1533] Furthermore, the emotion engine analyzes the user's facial expressions and voice to recognize their emotional state, and adjusts the visual and audio effects in the metaverse space in real time based on the recognized emotion. For example, if the user is emotional, the background music can be changed to a more uplifting one, or the lighting can be adjusted to be more vibrant.

[1534] Specific examples

[1535] Users take photos of new shoes and upload them to a server using a dedicated application. The server analyzes the received photos, generates a 3D model of the shoe, and stores it for users to access. Users can then put on a virtual reality headset and enter a virtual store to try on the generated 3D shoe model. The user's facial expressions and voice are analyzed by an emotion engine, which adjusts the lighting and background music in the store based on the emotion data.

[1536] Prompt Sentence Examples

[1537] "Generate a realistic 3D model based on a photograph of a new running shoe. The model should accurately reproduce the shoe's details and specific features (e.g., sole design, color tone)."

[1538] This allows users to experience a highly immersive metaverse space based on the photos and videos they take, and allows them to enjoy a real-time experience customized to their emotions at the time.

[1539] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1540] Step 1:

[1541] The user takes a photo or video.

[1542] Input: Photos and videos taken by users using devices such as smartphones and tablets.

[1543] Specific operation: The user launches the photo app and takes photos or videos while traveling or at an event. The captured data is saved in the device's local storage.

[1544] Output: Photo and video data saved in local storage.

[1545] Step 2:

[1546] The user uploads data.

[1547] Input: Photo and video data stored on the user's device.

[1548] How it works: The user opens the application and selects the photos and videos they have taken for uploading. The selected data is sent to the server as an HTTP POST request.

[1549] Output: Photo and video data sent to the server.

[1550] Step 3:

[1551] The server receives the data.

[1552] Input: HTTP POST request sent from the user's device.

[1553] Specific operation: The server receives the HTTP request and temporarily saves the sent photo or video files in the specified directory.

[1554] Output: Temporarily saved photo and video files.

[1555] Step 4:

[1556] The server analyzes the photos and videos.

[1557] Input: Temporarily saved photo and video files.

[1558] How it works: The server's image analysis module analyzes photo and video data using computer vision technologies such as OpenCV and TensorFlow, recognizing scenery, people, and movements, and extracting features from each frame.

[1559] Output: Analyzed feature information (scenery, people, movement) data.

[1560] Step 5:

[1561] The server uses generation AI to generate a three-dimensional metaverse space.

[1562] Input: Parsed feature information data.

[1563] Specific operation: The server uses generative AI, including deep learning models and 3D reconstruction algorithms, to generate a three-dimensional metaverse space based on the analysis results.

[1564] Output: Generated 3D metaverse spatial data.

[1565] Step 6:

[1566] The server stores and provides the generated three-dimensional metaverse space.

[1567] Input: Generated 3D Metaverse spatial data.

[1568] Specific operation: The server stores the generated 3D metaverse space data in a database, links the identifier to the user's profile, and provides the stored data for the user to access.

[1569] Output: Metaverse spatial data stored in a database.

[1570] Step 7:

[1571] Emotion data is acquired from the user's device.

[1572] Input: Facial expressions and voice captured by the user's device.

[1573] Specific operation: The user device captures facial expressions and voices during the user's experience and sends them to the emotion engine.

[1574] Output: Facial and speech data captured and sent to the emotion engine.

[1575] Step 8:

[1576] Adjusting the metaverse space in real time based on emotional data.

[1577] Input: Acquired emotion data.

[1578] Specific operation: The server analyzes the data sent from the emotion engine, recognizes the user's emotional state, and instantly adjusts the visual and sound effects in the metaverse space based on the recognized emotion.

[1579] Output: A real-time adjusted metaverse space.

[1580] Step 9:

[1581] A user accesses the metaverse space.

[1582] Input: User login information and metaverse space identifier.

[1583] Specific operation: The user puts on a head-mounted display or augmented reality device and launches a dedicated application. After logging in, the user selects a specific space from a list of saved metaverse spaces to begin the experience. The selected metaverse space data is then streamed in real time from the server and displayed to the user.

[1584] Output: A real-time display of the metaverse space experienced by the user.

[1585] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1586] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1587] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1588] [Fourth embodiment]

[1589] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1590] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1591] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1592] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1593] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1594] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1595] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1596] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1597] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1598] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1599] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1600] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1601] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1602] This invention is a system that recreates the scenery, people, and movements of a moment in a metaverse space based on photos and videos taken by the user. This system utilizes the user's device, a server, and generation AI to provide the user with an immersive experience.

[1603] System configuration

[1604] 1. User Device:

[1605] They act as devices that allow users to take photos and videos, including smartphones, tablets, and digital cameras.

[1606] A dedicated application is installed to allow users to send captured image data to a server.

[1607] Users access the saved metaverse space and use VR headsets or AR-enabled devices to experience it.

[1608] 2. Server:

[1609] It has a component that receives photo and video data sent from the user's device.

[1610] Computer vision techniques are used to analyze the received data and extract scenery, people, and movement.

[1611] Based on the extracted information, a 3D metaverse space is generated using generative AI.

[1612] The generated metaverse spatial data is stored in a database and provided for users to access.

[1613] 3. Generation AI:

[1614] Based on the analysis results, it includes deep learning models and 3D reconstruction algorithms to generate 3D models that reproduce real-world scenes, people, and movements.

[1615] System Operation

[1616] Data acquisition and transmission (user's device)

[1617] User takes a photo or video:

[1618] A user takes photos and videos using the device's camera app while traveling or at an event.

[1619] The captured data is stored in the device's local storage.

[1620] User uploads data:

[1621] The user opens the dedicated app and goes to the upload screen of the metaverse generation service.

[1622] Select the saved photo or video file and press the upload button.

[1623] The app sends the selected file to the server as an HTTP POST request.

[1624] Data reception and analysis (server)

[1625] Server receives data:

[1626] The server receives the HTTP request and retrieves the transmitted file data.

[1627] The received file is stored in a directory for temporary storage.

[1628] The server analyzes the photos and videos:

[1629] The server's image analysis module analyzes the transmitted photo and video data.

[1630] Image analysis uses computer vision libraries such as OpenCV and TensorFlow to recognize scenery and people.

[1631] In video analysis, a video is divided into frames, and each frame is subjected to image analysis to extract movement.

[1632] Generation and storage of 3D metaverse space (server)

[1633] The server generates a 3D metaverse space using generative AI:

[1634] The extracted feature information is input into a generation AI to generate a 3D metaverse space.

[1635] Generative AI uses GANs and 3D reconstruction algorithms to recreate realistic situations.

[1636] The server saves the generated metaverse space:

[1637] The generated 3D data is stored in a database along with metadata.

[1638] Generate an identifier (ID) for the stored spatial data and link it to the user's profile.

[1639] Providing a metaverse experience (user device)

[1640] User accesses the metaverse space:

[1641] The user puts on a VR headset or AR-compatible device and launches a dedicated app.

[1642] Log in to the server from the app and view a list of saved metaverse spaces.

[1643] Select a specific metaverse space from the list provided to begin the experience.

[1644] The app streams selected metaverse spatial data from the server and displays it to the user in real time.

[1645] Specific examples

[1646] 1. Relive your travel memories:

[1647] A user takes a photo or video in front of the Eiffel Tower while traveling in Paris (user's device).

[1648] The taken photo is uploaded by dragging and dropping it onto the upload screen of the metaverse generation service (user's device).

[1649] The server receives the photo data sent from the user's terminal and temporarily stores it (server).

[1650] The server's image analysis module recognizes the Eiffel Tower and the surrounding scenery from the sent photo and extracts its features (server).

[1651] The generative AI generates a 3D model of the Eiffel Tower based on the extracted features and recreates the background scenery (server).

[1652] The generated 3D model and scenery data are stored in a database and linked to the user's profile (server).

[1653] Users put on a VR headset at home and open a dedicated app to access the Eiffel Tower metaverse space (on their device).

[1654] Users can move freely within the metaverse and experience the Eiffel Tower and its surroundings in 360 degrees (on their device).

[1655] 2. Recreating child growth records:

[1656] A user records a video of their child's first birthday (on their device).

[1657] The captured video is uploaded to the metaverse generation service (user's device).

[1658] The server receives the video and analyzes the birthday cake and the child's movements (server).

[1659] Based on the analyzed information, the generation AI generates a 3D metaverse space for the birthday party (server).

[1660] The generated 3D data is stored in a database and made accessible to users (server).

[1661] The user uses a VR headset to relive their child's first birthday (on the user's device).

[1662] The system of the present invention allows users to realistically preserve and experience memories in a highly immersive alternative to two-dimensional photos and videos.

[1663] The processing flow will be explained below.

[1664] Step 1:

[1665] Data acquisition (user device)

[1666] Users use the camera app on their devices to take photos or videos of scenery at their travel destinations or events.

[1667] The captured data is stored in the device's local storage.

[1668] Step 2:

[1669] Data transmission (user's device)

[1670] Users open the dedicated app and select the photos and videos they have taken.

[1671] When you press the upload button, the app sends the selected data to the server as an HTTP POST request.

[1672] Step 3:

[1673] Data reception (server)

[1674] The server receives an HTTP POST request to retrieve the photo or video file.

[1675] To temporarily store the received data, save it in the specified directory.

[1676] Step 4:

[1677] Image and video analysis (server)

[1678] The server uses computer vision libraries (e.g., OpenCV and TensorFlow) to analyze the stored photo and video data.

[1679] In the case of images, the server extracts features such as scenery, people, and objects from the image.

[1680] In the case of video, the server divides the video into frames, analyzes each frame, and extracts movement.

[1681] Step 5:

[1682] 3D Metaverse Space Generation (Server)

[1683] Based on the analysis results, the server generates a 3D metaverse space using generative AI (e.g., GAN or 3D reconstruction algorithms).

[1684] Generative AI uses extracted features to recreate scenes, people, and movements.

[1685] Step 6:

[1686] Metaverse space storage (server)

[1687] The server stores the generated 3D metaverse spatial data in a database.

[1688] Generate an identifier (ID) for the saved space and link it to the user's profile.

[1689] Step 7:

[1690] Providing a metaverse experience (user device)

[1691] The user puts on a VR headset or AR-compatible device and launches a dedicated app.

[1692] Users log in to the server through the app and view a list of saved metaverse spaces.

[1693] Select a specific metaverse space from the list provided to begin the experience.

[1694] The app streams selected metaverse spatial data in real time from the server and displays it to the user.

[1695] In this way, users can experience an immersive three-dimensional metaverse space generated from two-dimensional photos and videos.

[1696] Example 1

[1697] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1698] Conventional recording methods using photographs and videos forced users to rely on two-dimensional information, making it difficult to recreate the atmosphere or three-dimensional experience of the scene. Furthermore, there were limited ways to preserve detailed memories of specific events or travel destinations and experience them later in real life. Furthermore, existing systems required complex processes for data analysis and reproduction, making practical application difficult.

[1699] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1700] In this invention, the server includes: means for receiving photos or videos acquired from a user terminal and analyzing scenery, people, and movements from the photos or videos; means for a generation AI to generate a 3D metaverse space based on the analysis results; means for saving the generated 3D metaverse space and providing it for user access; means for a user to take photos or videos and upload the data to the server using a dedicated application; means for the server to save the transmitted photos or videos in a temporary directory and analyze them using computer vision technology; and means for saving the generated metaverse space data in a database and linking the identifier to a user profile. This allows users to have a more immersive experience, enabling the realistic reproduction of on-site atmospheres and three-dimensional experiences.

[1701] "User terminal" refers to any digital device that allows a user to take and save photos and videos, and specific examples include smartphones and digital cameras.

[1702] A "server" refers to a computer system that receives, analyzes, and stores data sent from a user terminal, and provides various services to the user.

[1703] "Photos or videos" refers to still images or video data taken by a user using a digital device.

[1704] "Analysis" refers to the process of recognizing scenery, people, and movements from photos or videos and extracting their respective characteristics.

[1705] "Scenery" refers to the natural or man-made scenery or backgrounds contained in photographs or videos.

[1706] "People" refers to human figures appearing in photographs and videos.

[1707] "Motion" refers to the movement and activity of people and objects in a photo or video.

[1708] "Generative AI" refers to artificial intelligence techniques used to generate 3D Metaverse spaces based on analyzed data, including GANs and 3D reconstruction algorithms.

[1709] "3D Metaverse space" refers to a digital space in which real-world scenery, people, and movements are reproduced in three dimensions within a virtual space.

[1710] "Dedicated application" refers to software that users use to upload photos and videos to a server.

[1711] The "temporary directory" refers to a folder where the server temporarily stores data received.

[1712] "Computer vision technology" refers to the technology that allows computers to analyze images and videos and extract information. Examples include OpenCV and TensorFlow.

[1713] "Database" refers to a system for efficiently storing and managing the generated metaverse spatial data. A specific example is MongoDB.

[1714] "Identifier" refers to an ID that uniquely identifies stored metaverse spatial data.

[1715] "User profile" refers to a digital data structure for storing information and data related to a user.

[1716] "VR headset" refers to a device used by a user to experience a virtual reality space, including, for example, Oculus Rift.

[1717] "AR-enabled device" means a device used to experience an augmented reality space, including, for example, Microsoft HoloLens.

[1718] MODE FOR CARRYING OUT THE INVENTION

[1719] This invention is a system that recreates the scenery, people, and movements of a moment in a metaverse space based on photos and videos taken by the user. This system utilizes the user's device, a server, and generation AI to provide the user with an immersive experience.

[1720] 1. System Configuration

[1721] User terminal

[1722] A user device is a general term for a digital device that allows a user to take and save photos and videos, including smartphones and digital cameras. A dedicated application (e.g., the Metaverse Creator app) is installed on the device to transmit the captured data to a server. Users can access and experience the saved Metaverse space using a VR headset (e.g., Oculus Rift) or an AR-enabled device (e.g., Microsoft HoloLens).

[1723] server

[1724] The server receives photo and video data sent from user devices, analyzes it, and generates the metaverse space. The server includes the following main components:

[1725] Data receiving module: Receives HTTP requests and retrieves the transmitted file data.

[1726] Temporary storage: The directory where received data is temporarily stored (e.g. / tmp / uploads).

[1727] Image analysis module: Uses computer vision libraries such as OpenCV and TensorFlow to extract scenery, people, and movement from photo and video data.

[1728] Generative AI module: Uses GAN and 3D reconstruction algorithms to generate a 3D metaverse space based on extracted data.

[1729] Database: A database (e.g., MongoDB) to store the generated metaverse spatial data and associated metadata.

[1730] 2. System Operation

[1731] Data acquisition and transmission by the user's device

[1732] A user takes photos and videos using a smartphone or digital camera. For example, while traveling, the user takes a photo of the Eiffel Tower using the iPhone camera app and saves it to the camera roll. The user then launches the MetaverseCreator app, selects the photo, and uploads it. The app then sends the photo data to the server as an HTTP POST request.

[1733] Data reception and analysis by the server

[1734] The server receives the HTTP request and saves the data in a temporary directory. The image analysis module then analyzes the photo data, recognizing scenery such as the Eiffel Tower, people, and movements, and extracting feature points. This feature point information is stored as JSON data.

[1735] Creation and storage of 3D metaverse space

[1736] The server inputs the extracted feature information into a generative AI module, which then uses a GAN model and 3D reconstruction algorithm to generate a 3D structure of the Eiffel Tower and recreate the background scenery. The generated metaverse spatial data is stored in a database, and an identifier is generated and linked to the user's profile.

[1737] User experience of the metaverse space

[1738] Users put on a VR headset or AR-enabled device and launch the MetaverseViewer app. They log in within the app and view a list of Metaverse spaces linked to their profile. They select a specific Metaverse space and begin the experience. The app streams the Metaverse space data from the server and displays it on the user's device in real time.

[1739] Examples and prompts

[1740] For example, if a user uploads a photo of the Eiffel Tower taken during a trip to Paris, and a generative AI generates a 3D metaverse of the Eiffel Tower based on that photo, the actual prompt would look like this:

[1741] "Generate a 3D model of the Eiffel Tower based on the following photo."

[1742] "Reconstruct a scene from a child's birthday party using frames from this video."

[1743] The system of the present invention allows users to realistically preserve and experience memories in a highly immersive alternative to two-dimensional photos and videos.

[1744] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1745] Step 1:

[1746] A user takes a photo or video. A user takes a photo or video using a smartphone or digital camera. For example, while traveling, a user takes a photo of the Eiffel Tower using the camera app on their iPhone and saves it to their camera roll. The input is the camera app on the digital device, and the output is the photo or video data saved in local storage.

[1747] Step 2:

[1748] The user uploads data to the server using a dedicated application. The user launches the MetaverseCreator app, selects the photos and video files they have taken, and uploads them. The input is the photo or video file saved in local storage, and the output is the file data sent as an HTTP POST request. During this process, an upload progress bar is displayed.

[1749] Step 3:

[1750] The server receives the data. The server receives the HTTP request and saves the submitted file data and metadata in a temporary directory (e.g., / tmp / uploads). The input is the photo or video data included in the HTTP request body, and the output is the file saved in the temporary directory.

[1751] Step 4:

[1752] The server analyzes photos and videos. The server's image analysis module (e.g., OpenCV, TensorFlow) analyzes the photo and video data stored in a temporary directory. The input is the file stored in the temporary directory, and the output is feature information about the scenery, people, and actions stored in JSON format. Specifically, feature points of the Eiffel Tower and people are extracted from photos, and each frame of the video is analyzed to detect movement.

[1753] Step 5:

[1754] The server uses generative AI to generate a 3D metaverse space. The server inputs the extracted feature point information into a generative AI module (e.g., GAN or 3D reconstruction algorithm) to generate the 3D metaverse space. The input is JSON data of the feature point information, and the output is data in the generated 3D metaverse space. For example, a GAN model generates the 3D structure of the Eiffel Tower and recreates the background scenery.

[1755] Step 6:

[1756] The server saves the generated metaverse space in a database. The server saves the generated metaverse space data in a database (e.g., MongoDB) and links the identifier to the user profile. The input is the 3D metaverse space data and the user ID, and the output is the data saved in the database and the generated identifier.

[1757] Step 7:

[1758] A user accesses the Metaverse space by putting on a VR headset (e.g., Oculus Rift) or an AR-enabled device (e.g., Microsoft HoloLens) and launching the MetaverseViewer app. The input is the application on the device, and the output is the Metaverse space data streamed from the server. When the user selects a Metaverse space in the app and presses the "Start Experience" button, the app displays the data in real time, and the user enjoys an immersive experience.

[1759] (Application example 1)

[1760] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1761] There is a need for a method that allows users to upload photos and videos they have taken to the metaverse space through a dedicated application, and then share the 3D metaverse space generated from it with other users, thereby providing a visually immersive experience. However, existing technologies do not easily enable the generation and sharing of such advanced metaverse spaces, so a new solution is needed.

[1762] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1763] In this invention, the server includes means for receiving photos or videos acquired from a user terminal and analyzing scenery, people, and movements from the photos or videos, means for a generation AI to generate a 3D metaverse space based on the analysis results, means for saving the generated 3D metaverse space and providing it for users to access, means for users to upload and share photos or videos using a smartphone application, and means for sharing the generated 3D metaverse space with friends and family. This allows users to generate realistic 3D metaverse spaces based on photos and videos they have taken and easily share them with other users, providing a visually realistic and immersive experience.

[1764] "User terminal" refers to a device that takes photos and videos and has installed an application for sending that data to a server.

[1765] A "server" refers to a computer system that receives data sent from a user terminal, analyzes it, and performs the necessary processing.

[1766] "Generative AI" refers to artificial intelligence technology for generating a three-dimensional metaverse space based on the analysis of photos and videos sent from user devices.

[1767] "3D Metaverse space" refers to a three-dimensional virtual space generated based on real-world scenery, people, and movements.

[1768] "Smartphone application" refers to software that is installed on a smartphone and allows users to upload photos and videos, share and experience the metaverse space.

[1769] "Analyzing" refers to the process of extracting specific information (scenery, people, movements, etc.) from photos and videos.

[1770] "Saving" refers to the act of storing the generated 3D metaverse spatial data in a state that allows it to be accessed later.

[1771] "Providing" refers to the process of allowing a user to access a saved three-dimensional metaverse space.

[1772] "Sharing" refers to the act of exchanging the generated 3D metaverse space with other users in an easily accessible manner.

[1773] "Friends and Family" refers to close people with whom the user wants to share.

[1774] This invention is a system that recreates the scenery, people, and movements of a moment in a 3D metaverse space based on photos and videos taken by the user. This system utilizes the user's device, a server, and a generation AI to provide the user with an immersive experience.

[1775] System configuration

[1776] It consists of the following main components:

[1777] 1. User Device:

[1778] Camera device: A device that allows a user to take photos or videos. Examples include smartphones, tablets, and digital cameras.

[1779] Dedicated application: An application is installed to send the captured data to the server. VR headsets and AR-compatible devices are also used to access and experience the generated 3D metaverse space.

[1780] 2. Server:

[1781] Receiving module: Contains a component that receives and temporarily stores photo and video data sent from the user device.

[1782] Image analysis module: Analyzes received photo and video data using computer vision technology, specifically using libraries such as OpenCV and TensorFlow.

[1783] Generative AI module: Generates a 3D metaverse space based on the extracted information. This uses deep learning models, including GANs and 3D reconstruction algorithms.

[1784] Database: Stores the generated 3D metaverse spatial data and makes it accessible to users.

[1785] 3. Generation AI:

[1786] Deep learning model: A technology for generating 3D models that reproduce real-world scenery, people, and movements based on analysis results sent from the user's device.

[1787] 3D reconstruction algorithm: Creates a realistic 3D space based on the analysis results.

[1788] System Operation

[1789] 1. Data acquisition and transmission (user terminal):

[1790] Users take photos and videos using their smartphones and upload the data to the server via a dedicated application. Uploads are made via HTTP POST requests.

[1791] 2. Data reception and analysis (server):

[1792] The server receives and temporarily stores the photos and video data sent from the user's device. It then uses computer vision technology to analyze the scenery, people, and movements from the received data. Libraries such as OpenCV and TensorFlow are used for this.

[1793] 3. Generation and storage of 3D metaverse space (server):

[1794] The analysis results are input into the generation AI to generate a 3D metaverse space. The generated 3D data is stored in a database and made available for users to access.

[1795] 4. Providing a Metaverse experience (user device):

[1796] Users can access the generated 3D metaverse space through a dedicated application using a VR headset or AR-compatible device, and can also share the generated metaverse space with friends and family.

[1797] Specific examples

[1798] Travel Reenactment:

[1799] Users upload photos taken in front of the Eiffel Tower during their travels. The server analyzes the photos and uses generative AI to generate a 3D metaverse of the Eiffel Tower and its surroundings. Users can access this metaverse by wearing a VR headset and enjoy a 360-degree experience.

[1800] Recreating a child's growth record:

[1801] Users upload videos of their child's first birthday. The server analyzes the videos and uses generative AI to generate a 3D space of the birthday party. Users can access this recreated space and relive their child's growth and memories.

[1802] Prompt Sentence Examples

[1803] "Upload a photo of the Eiffel Tower and generate a realistic 3D metaverse space based on this photo."

[1804] This system allows users to easily recreate their own photos and videos in a 3D metaverse space and share that space with others, providing a visually immersive experience.

[1805] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1806] Step 1:

[1807] The user takes a photo or video

[1808] Users take photos and videos using the camera app on their smartphones.

[1809] Input: Photo or video data from a smartphone camera app

[1810] Output: Photo or video files stored in the smartphone's local storage

[1811] Step 2:

[1812] User uploads data

[1813] The user opens the application, selects the photos and video files they have taken, and presses the upload button, which sends these files to the server as an HTTP POST request.

[1814] Input: Photo or video files selected through a dedicated application, server upload endpoint

[1815] Output: HTTP POST request sent to the server

[1816] Step 3:

[1817] The server receives the data

[1818] The server receives the HTTP POST request, retrieves the sent file data, and saves it in a temporary directory.

[1819] Input: Photo or video data received via an HTTP POST request

[1820] Output: File data saved in temporary directory

[1821] Step 4:

[1822] The server analyzes the photos and videos.

[1823] The server's image analysis module uses OpenCV and TensorFlow to analyze the received photo and video data. Image analysis recognizes scenery and people, while video analysis divides the video into frames to extract movement.

[1824] Input: Photo or video data saved in the temporary directory

[1825] Output: Scenery, people, and movement information as analysis results

[1826] Step 5:

[1827] The server generates a 3D metaverse space using generation AI.

[1828] The server inputs the analysis results into the generation AI, which then generates a 3D metaverse space using GAN and 3D reconstruction algorithms.

[1829] Input: Scenery, people, and movement information as analysis results

[1830] Output: Generated 3D metaverse spatial data

[1831] Step 6:

[1832] The server stores the generated 3D metaverse spatial data.

[1833] The server stores the generated 3D metaverse spatial data in a database and makes it accessible to users.

[1834] Input: Generated 3D metaverse spatial data, user identification information

[1835] Output: 3D metaverse spatial data stored in a database, metaverse space linked to user profile

[1836] Step 7:

[1837] A user accesses the metaverse space

[1838] Users use a VR headset or AR-compatible device to log in through a dedicated application, select one from a list of saved 3D metaverse spaces, and begin the experience.

[1839] Input: User login information, list of 3D metaverse spaces stored in the database

[1840] Output: 3D metaverse space streamed to user's device

[1841] Step 8:

[1842] Users share the metaverse space

[1843] Users can share the 3D metaverse space they create with friends and family through a dedicated application, generating a shared link and granting access rights to specific users.

[1844] Input: 3D metaverse space linked to user profile, shared identity information

[1845] Output: Shared link and access rights to the shared 3D metaverse space

[1846] The above are the specific processing steps in the system for implementing this invention. At each step, various data processing and calculations are performed based on the input data, and the results are passed on to the next step, thereby operating the entire system.

[1847] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1848] The present invention is a system that recreates the scenery, people, and movements of a user in a metaverse space based on photos and videos taken by the user, and further adjusts the experience by recognizing the user's emotions. The following describes the detailed operation of the system.

[1849] System configuration

[1850] 1. User Device:

[1851] They act as devices that allow users to take photos and videos, including smartphones, tablets, and digital cameras.

[1852] A dedicated application is installed to allow users to send captured image data to a server.

[1853] It also has the function of capturing the user's facial expressions and voice and sending them to the emotion engine.

[1854] Users access the saved metaverse space and use VR headsets or AR-enabled devices to experience it.

[1855] 2. Server:

[1856] It has a component that receives photo and video data sent from the user's device, as well as emotion data recognized by the emotion engine.

[1857] Computer vision techniques are used to analyze the received data and extract scenery, people, and movement.

[1858] Based on the extracted information, a 3D metaverse space is generated using generative AI.

[1859] The generated metaverse spatial data is stored in a database and provided for users to access.

[1860] Tailor your metaverse experience in real time based on data from the emotion engine.

[1861] 3. Generation AI:

[1862] Based on the analysis results, it includes deep learning models and 3D reconstruction algorithms to generate 3D models that reproduce real-world scenes, people, and movements.

[1863] 4. Emotion Engine:

[1864] Recognizes emotions by analyzing the user's facial expressions and voice.

[1865] Adjust visual and audio effects within the metaverse space based on the emotions recognized.

[1866] System Operation

[1867] Data acquisition and transmission (user's device)

[1868] User takes a photo or video:

[1869] A user takes photos and videos using the device's camera app while traveling or at an event.

[1870] The captured data is stored in the device's local storage.

[1871] User uploads data:

[1872] The user opens the dedicated app and selects the photos and videos they have taken.

[1873] When you press the upload button, the app sends the selected data to the server as an HTTP POST request.

[1874] Data reception and analysis (server)

[1875] Server receives data:

[1876] The server receives an HTTP POST request to retrieve the photo or video file.

[1877] To temporarily store the received data, save it in the specified directory.

[1878] The server analyzes the photos and videos:

[1879] The server's image analysis module analyzes the transmitted photo and video data.

[1880] Image analysis uses computer vision libraries such as OpenCV and TensorFlow to recognize scenery and people.

[1881] In video analysis, a video is divided into frames, and each frame is subjected to image analysis to extract movement.

[1882] 3D Metaverse Space Generation (Server)

[1883] The server generates a 3D metaverse space using generative AI:

[1884] The extracted feature information is input into a generation AI to generate a 3D metaverse space.

[1885] Generative AI uses GANs and 3D reconstruction algorithms to recreate realistic situations.

[1886] Metaverse space storage (server)

[1887] The server stores the generated 3D metaverse space:

[1888] The server stores the generated 3D metaverse spatial data in a database.

[1889] Generate an identifier (ID) for the stored spatial data and link it to the user's profile.

[1890] Emotion recognition and user experience adjustment (emotion engine and server)

[1891] Emotion engine recognizes emotions:

[1892] The user device captures the user's facial expressions and voice and sends them to the emotion engine.

[1893] The emotion engine analyzes this data and recognizes the user's emotional state.

[1894] Emotion-based experience tailoring:

[1895] The emotion data obtained from the emotion engine is received by the server.

[1896] The server changes the visual and sound effects in the metaverse space in real time based on the emotional data.

[1897] For example, if the user is impressed, the scenery will be made more vivid and the background music will be changed to something more exciting.

[1898] Providing a metaverse experience (user device)

[1899] User accesses the metaverse space:

[1900] The user puts on a VR headset or AR-compatible device and launches a dedicated app.

[1901] Users log in to the server through the app and view a list of saved metaverse spaces.

[1902] Select a specific metaverse space from the list provided to begin the experience.

[1903] The app streams selected metaverse spatial data in real time from the server and displays it to the user.

[1904] Specific examples

[1905] 1. Travel memory recall and emotional regulation:

[1906] A user takes a photo or video in front of the Eiffel Tower while traveling in Paris (user's device).

[1907] The taken photo is uploaded by dragging and dropping it onto the upload screen of the metaverse generation service (user's device).

[1908] The server receives the photo data sent from the user's device and temporarily stores it (server).

[1909] The server's image analysis module recognizes the Eiffel Tower and the surrounding scenery from the sent photo and extracts its features (server).

[1910] The generative AI generates a 3D model of the Eiffel Tower based on the extracted features and recreates the background scenery (server).

[1911] The generated 3D model and scenery data are stored in a database and linked to the user's profile (server).

[1912] Users put on a VR headset at home and open a dedicated app to access the Eiffel Tower metaverse space (on their device).

[1913] While the user is experiencing the experience, the emotion engine analyzes the user's facial expressions and if it recognizes emotion, the server adjusts the effects and changes the background music to an emotional one (server).

[1914] 2. Recapitulation of child development records and emotional regulation:

[1915] A user records a video of their child's first birthday (on their device).

[1916] The captured video is uploaded to the metaverse generation service (user's device).

[1917] The server receives the video and analyzes the birthday cake and the child's movements (server).

[1918] Based on the analyzed information, the generation AI generates a 3D metaverse space for the birthday party (server).

[1919] The generated 3D data is stored in a database and made accessible to users (server).

[1920] The user uses a VR headset to relive their child's first birthday (on the user's device).

[1921] If the emotion engine recognizes the user's joy, the server will brighten the lights in the space and make the background music more enjoyable (server).

[1922] The system of the present invention allows users to realistically preserve memories in a highly immersive alternative to two-dimensional photos and videos, and allows for customized experiences that reflect their emotional state at any given time.

[1923] The processing flow will be explained below.

[1924] Step 1:

[1925] Data acquisition (user device)

[1926] Users use the camera app on their devices to take photos or videos of scenery at their travel destinations or events.

[1927] The captured data is stored in the device's local storage.

[1928] Step 2:

[1929] Data transmission (user's device)

[1930] Users open the dedicated app and select the photos and videos they have taken.

[1931] When you press the upload button, the app sends the selected data to the server as an HTTP POST request.

[1932] Step 3:

[1933] Data reception (server)

[1934] The server receives an HTTP POST request to retrieve the photo or video file.

[1935] To temporarily store the received data, save it in the specified directory.

[1936] Step 4:

[1937] Image and video analysis (server)

[1938] The server uses computer vision libraries (e.g., OpenCV and TensorFlow) to analyze the stored photo and video data.

[1939] In the case of images, the server extracts features such as scenery, people, and objects from the image.

[1940] In the case of video, the server divides the video into frames, analyzes each frame, and extracts movement.

[1941] Step 5:

[1942] 3D Metaverse Space Generation (Server)

[1943] Based on the analysis results, the server generates a 3D metaverse space using generative AI (e.g., GAN or 3D reconstruction algorithms).

[1944] Generative AI uses extracted features to recreate scenes, people, and movements.

[1945] Step 6:

[1946] Metaverse space storage (server)

[1947] The server stores the generated 3D metaverse spatial data in a database.

[1948] Generate an identifier (ID) for the saved space and link it to the user's profile.

[1949] Step 7:

[1950] Data acquisition for emotion recognition (user device)

[1951] The user puts on a VR headset or AR-compatible device and launches the dedicated app.

[1952] Sensors and microphones begin to operate to capture the user's facial expressions and voice.

[1953] Step 8:

[1954] Sending emotion data (user device)

[1955] The user's facial expression and voice data are sent to the server in real time.

[1956] This data is then forwarded to a dedicated endpoint to be analyzed by the emotion engine.

[1957] Step 9:

[1958] Receiving and analyzing emotion data (server and emotion engine)

[1959] The server passes the received facial expression and voice data to the emotion engine.

[1960] The emotion engine analyzes this data and recognizes the user's emotional state.

[1961] Step 10:

[1962] Emotion-based experience adjustment (server)

[1963] The server adjusts the visual and sound effects in the metaverse space in real time based on the emotional state data obtained from the emotion engine.

[1964] For example, if the user is moved, the scenery is made more vivid and the background music is changed to something more moving.

[1965] Step 11:

[1966] Providing a metaverse experience (user device)

[1967] Users log in to the server through a dedicated app and view a list of saved metaverse spaces.

[1968] Select a specific metaverse space from the list provided to begin the experience.

[1969] The app streams selected metaverse spatial data in real time from the server and displays it to the user.

[1970] In this way, users can experience an immersive three-dimensional metaverse space generated from two-dimensional photos and videos, and further customize it to suit their emotions at the time.

[1971] Example 2

[1972] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1973] Conventional methods simply record photos and videos, but have the problem of not being able to reflect or reproduce the user's emotions and experiences in real time. Furthermore, simple 2D data lacks a sense of immersion, making it difficult to reproduce the emotions and joy that users actually felt at the time.

[1974] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1975] In this invention, the server includes means for receiving photos or videos acquired from a user terminal and analyzing scenery, people, and movements from the photos or videos, means for a generation AI to generate a 3D metaverse space based on the analysis results, means for saving the generated 3D metaverse space and providing it for user access, means for capturing the user's facial expressions and voice and transmitting them to an emotion engine that recognizes emotions, and means for adjusting visual and audio effects in the metaverse space in real time based on the recognized emotion data. This allows users to realistically recreate past experiences with a high level of immersion and have a customized experience that matches their emotions at the time.

[1976] "User terminal" refers to a device that a user uses to take photos and videos and upload the data to a server, and includes smartphones, tablets, digital cameras, etc.

[1977] "Server" refers to a computer system that receives data sent from a user terminal, analyzes, processes, stores, and provides the data.

[1978] "Photos or videos" refers to still images or video files taken by a user using a device.

[1979] "Analysis of scenery, people, and movements" refers to the process of using computer vision technology to recognize scenery, people, and movements in photographs or video data and extract their characteristics.

[1980] "Generative AI" refers to deep learning models and 3D reconstruction algorithms for generating a 3D metaverse space based on extracted feature information.

[1981] "3D Metaverse space" refers to a three-dimensional virtual space generated by a generation AI to recreate real-world situations as virtual environments.

[1982] An "emotion engine" refers to software or hardware that analyzes a user's facial expressions and voice and recognizes the user's emotional state.

[1983] "Real-time adjustment of visual and audio effects" refers to the process of dynamically changing the images and audio displayed within the metaverse space to match the user's emotions based on recognized user emotional data.

[1984] "VR headset" refers to a device worn on the head to immerse a user in a virtual reality environment.

[1985] "AR-compatible device" refers to a device that uses augmented reality technology to overlay virtual objects on the real world.

[1986] "Computer vision technology" refers to technology that enables computers to analyze and understand the content of images and videos, and includes libraries such as OpenCV and TensorFlow.

[1987] This invention is a system that recreates the scenery, people, and movements of a moment in a 3D metaverse space based on photos and videos taken on a user's device, and adjusts the experience by recognizing the user's emotions. The detailed operation of the system is explained below.

[1988] A user takes a photo or video using a user device such as a smartphone, tablet, or digital camera. This data is temporarily stored on the user's device. The user then uses a dedicated application to select the photos or videos they have taken and upload them to the server. Uploads are made as HTTP POST requests.

[1989] The server temporarily stores the received photo or video data in a specified directory. The server's image analysis module then analyzes the data to recognize and extract scenery, people, and movements. This analysis uses computer vision technologies such as OpenCV and TensorFlow.

[1990] Based on the analysis results, the server uses generative AI to generate a 3D metaverse space. This generative AI uses GAN (generative adversarial networks) and 3D reconstruction algorithms to create a realistic virtual space. The generated 3D metaverse space data is then stored in a database, and an identifier is generated and linked to the user's profile.

[1991] Users put on a VR headset or AR-compatible device and log in to the server through a dedicated application. A list of saved 3D metaverse spaces is displayed, and the user selects a specific space to begin the experience. The selected space data is streamed in real time by the server and displayed on the user's device.

[1992] During the experience, the user's device captures facial expressions and voice and sends this data to the emotion engine. The emotion engine analyzes the user's emotional state and sends the recognized emotional data to the server. The server uses this data to adjust the visual and audio effects in the metaverse space in real time. For example, if the user is emotional, the scenery will be made more vivid and the background music will be more exciting.

[1993] For example, a user takes a photo in front of the Eiffel Tower while traveling in Paris and uploads the photo data to a server using a dedicated app. Based on this, the server generates a 3D metaverse space, which the user can access. The emotion engine recognizes the user's emotions during the experience, and the server adjusts the background music accordingly.

[1994] An example of a prompt sentence is as follows:

[1995] "Generate and display a 3D metaverse of the Eiffel Tower based on photos from a trip to Paris."

[1996] In this way, the system of the present invention allows users to realistically preserve memories in a highly immersive alternative to two-dimensional photos and videos, and allows for customized experiences based on emotional state.

[1997] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1998] Step 1:

[1999] The user takes a photo or video.

[2000] Action: The user takes a photo using a smartphone or digital camera.

[2001] Input: Real-world scenes and people.

[2002] Output: Captured photo or video files (formats such as JPEG or MP4).

[2003] Step 2:

[2004] The user uploads the captured data to the server.

[2005] How it works: Use the dedicated app to select the shooting data and press the upload button.

[2006] Input: The captured photo or video file.

[2007] Output: Sent as an HTTP POST request to the server.

[2008] Step 3:

[2009] The server receives and stores the data.

[2010] How it works: The server receives an HTTP POST request and temporarily saves the photos and videos to the specified directory.

[2011] Input: The photo or video file included in the HTTP POST request.

[2012] Output: Temporarily saved photo or video files.

[2013] Step 4:

[2014] The server analyzes the photos and videos.

[2015] How it works: The server's image analysis module uses OpenCV and TensorFlow to analyze photos and videos, recognizing scenery and people, and dividing videos into frames to extract movement.

[2016] Input: A saved photo or video file.

[2017] Output: Recognized scenery, people, and movement feature information.

[2018] Step 5:

[2019] The server generation AI generates a 3D metaverse space.

[2020] How it works: The extracted feature information is input into a generative AI, which then uses GAN and 3D reconstruction algorithms to generate a 3D metaverse space.

[2021] Input: Extracted feature information.

[2022] Output: The generated 3D metaverse spatial data.

[2023] Step 6:

[2024] Stores metaverse spatial data in a database.

[2025] How it works: The generated 3D Metaverse spatial data is stored in a database, and an identifier is generated and linked to the user's profile.

[2026] Input: Generated 3D metaverse spatial data.

[2027] Output: 3D metaverse spatial data and identifiers stored in a database.

[2028] Step 7:

[2029] A user accesses the metaverse space.

[2030] How it works: A user puts on a VR headset or AR-enabled device and logs into the server through a dedicated app. A list of saved Metaverse spaces is displayed, and the user can select a specific experience. The selected space data is streamed in real time.

[2031] Input: Login information to the server, identifier of the selected metaverse space.

[2032] Output: Metaverse spatial footage streamed in real time to the user's device.

[2033] Step 8:

[2034] The user's emotions are captured and the server tailors the experience.

[2035] How it works: The user's device captures facial expressions and voice and sends them to the emotion engine. The emotion engine analyzes the emotional data and sends it to the server. The server uses this data to adjust the visual and audio effects in the metaverse space in real time.

[2036] Input: Captured facial and voice data, analyzed emotion data.

[2037] Output: A metaverse spatial experience with visual and audio effects tuned in real time.

[2038] (Application example 2)

[2039] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2040] In modern society, taking photos and videos has become an important means of preserving and re-experiencing memories. However, it is difficult to fully reproduce the emotions and atmosphere of a moment with two-dimensional photos and videos, and they are unable to fully recreate the excitement and joy felt by the user. It is also difficult for a user to relive an experience captured at a specific location with a sense of presence from another location. Furthermore, no system has been available to date that can provide a customized experience based on the user's emotions. There is a need for a system that can solve these issues and provide a more immersive and emotionally responsive real-time experience.

[2041] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2042] In this invention, the server includes means for receiving photos or videos acquired from a user device and analyzing scenery, people, and movements from the photos or videos, means for a generation AI to generate a 3D metaverse space based on the analysis results, means for saving the generated 3D metaverse space and providing it for user access, means for acquiring emotional data from the user device, and means for adjusting visual and audio effects in the metaverse space in real time based on the acquired emotional data. This allows users to experience a highly immersive metaverse space based on the photos and videos they have taken, and allows them to enjoy a real-time experience customized to their emotions at the time.

[2043] A "user terminal" is a device that a user uses to take photos and videos and send the data to a server, and includes smartphones, tablets, digital cameras, etc.

[2044] "Server" refers to a computer system that receives and analyzes data sent from user terminals, and generates, stores, and provides the metaverse space.

[2045] "Generative AI" is a technology that includes deep learning models and 3D reconstruction algorithms for generating a three-dimensional metaverse space based on data received from user devices.

[2046] "Emotion data" is information about the emotional state obtained by analyzing the user's facial expressions and voice.

[2047] "Means for real-time adjustment" refers to technology that instantly adjusts visual and sound effects within the metaverse space based on acquired emotional data.

[2048] A "metaverse space" is a three-dimensional virtual space where users can experience virtual reality.

[2049] "Visual and audio effects" refers to visual and audio effects that enhance the user experience, such as adjusting lighting or changing background music.

[2050] This invention is a system that recreates the scenery, people, and movements of a moment in a three-dimensional metaverse space based on photos and videos taken by the user, and further adjusts the experience by recognizing the user's emotions. Detailed embodiments for carrying out this invention are described below.

[2051] First, a user terminal is a device that includes a smartphone, tablet, digital camera, etc., that allows the user to take photos and videos and send the data to a server. When a user takes a photo or video, the data is sent to the server through a dedicated application. At this stage, the user's facial expressions and voice are also captured and sent as emotion data.

[2052] The server receives photo, video, and emotion data sent from the user's device. It analyzes the received data and uses computer vision technology to extract scenery, people, and movements. Specifically, image analysis is performed using libraries such as OpenCV and TensorFlow. Based on the analysis results, a generative AI generates a 3D metaverse space. The generative AI includes deep learning models and 3D reconstruction algorithms.

[2053] The generated metaverse space is stored on a server and made accessible to users. Users can access the generated metaverse space and experience it with a high level of immersion using a virtual reality headset or an augmented reality-enabled device.

[2054] Furthermore, the emotion engine analyzes the user's facial expressions and voice to recognize their emotional state, and adjusts the visual and audio effects in the metaverse space in real time based on the recognized emotion. For example, if the user is emotional, the background music can be changed to a more uplifting one, or the lighting can be adjusted to be more vibrant.

[2055] Specific examples

[2056] Users take photos of new shoes and upload them to a server using a dedicated application. The server analyzes the received photos, generates a 3D model of the shoe, and stores it for users to access. Users can then put on a virtual reality headset and enter a virtual store to try on the generated 3D shoe model. The user's facial expressions and voice are analyzed by an emotion engine, which adjusts the lighting and background music in the store based on the emotion data.

[2057] Prompt Sentence Examples

[2058] "Generate a realistic 3D model based on a photograph of a new running shoe. The model should accurately reproduce the shoe's details and specific features (e.g., sole design, color tone)."

[2059] This allows users to experience a highly immersive metaverse space based on the photos and videos they take, and allows them to enjoy a real-time experience customized to their emotions at the time.

[2060] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2061] Step 1:

[2062] The user takes a photo or video.

[2063] Input: Photos and videos taken by users using devices such as smartphones and tablets.

[2064] Specific operation: The user launches the photo app and takes photos or videos while traveling or at an event. The captured data is saved in the device's local storage.

[2065] Output: Photo and video data saved in local storage.

[2066] Step 2:

[2067] The user uploads data.

[2068] Input: Photo and video data stored on the user's device.

[2069] How it works: The user opens the application and selects the photos and videos they have taken for uploading. The selected data is sent to the server as an HTTP POST request.

[2070] Output: Photo and video data sent to the server.

[2071] Step 3:

[2072] The server receives the data.

[2073] Input: HTTP POST request sent from the user's device.

[2074] Specific operation: The server receives the HTTP request and temporarily saves the sent photo or video files in the specified directory.

[2075] Output: Temporarily saved photo and video files.

[2076] Step 4:

[2077] The server analyzes the photos and videos.

[2078] Input: Temporarily saved photo and video files.

[2079] How it works: The server's image analysis module analyzes photo and video data using computer vision technologies such as OpenCV and TensorFlow, recognizing scenery, people, and movements, and extracting features from each frame.

[2080] Output: Analyzed feature information (scenery, people, movement) data.

[2081] Step 5:

[2082] The server uses generation AI to generate a three-dimensional metaverse space.

[2083] Input: Parsed feature information data.

[2084] Specific operation: The server uses generative AI, including deep learning models and 3D reconstruction algorithms, to generate a three-dimensional metaverse space based on the analysis results.

[2085] Output: Generated 3D metaverse spatial data.

[2086] Step 6:

[2087] The server stores and provides the generated three-dimensional metaverse space.

[2088] Input: Generated 3D Metaverse spatial data.

[2089] Specific operation: The server stores the generated 3D metaverse space data in a database, links the identifier to the user's profile, and provides the stored data for the user to access.

[2090] Output: Metaverse spatial data stored in a database.

[2091] Step 7:

[2092] Emotion data is acquired from the user's device.

[2093] Input: Facial expressions and voice captured by the user's device.

[2094] Specific operation: The user device captures facial expressions and voices during the user's experience and sends them to the emotion engine.

[2095] Output: Facial and speech data captured and sent to the emotion engine.

[2096] Step 8:

[2097] Adjusting the metaverse space in real time based on emotional data.

[2098] Input: Acquired emotion data.

[2099] Specific operation: The server analyzes the data sent from the emotion engine, recognizes the user's emotional state, and instantly adjusts the visual and sound effects in the metaverse space based on the recognized emotion.

[2100] Output: A real-time adjusted metaverse space.

[2101] Step 9:

[2102] A user accesses the metaverse space.

[2103] Input: User login information and metaverse space identifier.

[2104] Specific operation: The user puts on a head-mounted display or augmented reality device and launches a dedicated application. After logging in, the user selects a specific space from a list of saved metaverse spaces to begin the experience. The selected metaverse space data is then streamed in real time from the server and displayed to the user.

[2105] Output: A real-time display of the metaverse space experienced by the user.

[2106] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2107] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2108] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2109] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2110] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2111] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2112] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2113] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2114] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2115] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2116] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2117] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2118] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2119] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2120] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2121] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2122] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2123] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2124] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2125] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2126] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2127] The following is further disclosed regarding the above embodiment.

[2128] (Claim 1)

[2129] Photos or videos acquired from the user's device

[2130] A server receives the photograph or video and analyzes the scenery, people, and movements of the photograph or video;

[2131] A means for generating a three-dimensional metaverse space by a generation AI based on the analysis results;

[2132] A means for storing the generated 3D metaverse space and providing it so that users can access it;

[2133] A system including:

[2134] (Claim 2)

[2135] The system of claim 1, wherein the generated three-dimensional metaverse space is experienced by a user using a VR headset and an AR-enabled device.

[2136] (Claim 3)

[2137] The system of claim 1, wherein the server uses computer vision technology to analyze the photos or videos.

[2138] "Example 1"

[2139] (Claim 1)

[2140] Photos or videos acquired from the user's device

[2141] A server receives the photograph or video and analyzes the scenery, people, and movements of the photograph or video;

[2142] A means for generating a three-dimensional metaverse space by a generation AI based on the analysis results;

[2143] A means for storing the generated 3D metaverse space and providing it so that users can access it;

[2144] A means for users to take photos and videos and upload the data to a server using a dedicated application;

[2145] The server stores the transmitted photos and videos in a temporary directory and analyzes them using computer vision technology.

[2146] means for storing the generated metaverse spatial data in a database and linking the identifier to a user profile;

[2147] A system including:

[2148] (Claim 2)

[2149] The system of claim 1, further comprising a user experiencing the generated three-dimensional metaverse space using a VR headset or an AR-enabled device.

[2150] (Claim 3)

[2151] The system of claim 1, characterized in that the server uses computer vision technology and deep learning models to analyze the photos or videos.

[2152] "Application Example 1"

[2153] (Claim 1)

[2154] Photos or videos acquired from the user's device

[2155] A server receives the photograph or video and analyzes the scenery, people, and movements of the photograph or video;

[2156] A means for generating a three-dimensional metaverse space by a generation AI based on the analysis results;

[2157] A means for storing the generated 3D metaverse space and providing it so that users can access it;

[2158] A means for users to upload and share photos or videos using a smartphone application;

[2159] A way to share the generated 3D metaverse space with friends and family,

[2160] A system including:

[2161] (Claim 2)

[2162] The system of claim 1, wherein the generated three-dimensional metaverse space is experienced by a user using a VR headset and an AR-enabled device.

[2163] (Claim 3)

[2164] The system of claim 1, wherein the server uses computer vision technology to analyze the photos or videos.

[2165] "Example 2: Combining Emotion Engines"

[2166] (Claim 1)

[2167] Photos or videos acquired from the user's device

[2168] A server receives the photograph or video and analyzes the scenery, people, and movements of the photograph or video;

[2169] A means for generating a three-dimensional metaverse space by a generation AI based on the analysis results;

[2170] A means for storing the generated 3D metaverse space and providing it so that users can access it;

[2171] means for capturing a user's facial expressions and voice and transmitting them to an emotion engine for emotion recognition;

[2172] A means for adjusting visual and audio effects in the metaverse space in real time based on the recognized emotional data; and

[2173] A system including:

[2174] (Claim 2)

[2175] The system of claim 1, wherein the generated three-dimensional metaverse space is experienced by a user using a VR headset and an AR-enabled device.

[2176] (Claim 3)

[2177] The system of claim 1, wherein the server uses computer vision technology to analyze the photos or videos.

[2178] "Application example 2 when combining emotion engines"

[2179] (Claim 1)

[2180] Photos or videos acquired from the user's device

[2181] A server receives the photograph or video and analyzes the scenery, people, and movements of the photograph or video;

[2182] A means for generating a three-dimensional metaverse space by a generation AI based on the analysis results;

[2183] A means for storing the generated 3D metaverse space and providing it so that users can access it;

[2184] A means for acquiring emotion data from a user's terminal;

[2185] a means for adjusting visual and audio effects in the metaverse space in real time based on the acquired emotional data;

[2186] A system including:

[2187] (Claim 2)

[2188] 10. The system of claim 1, wherein the generated three-dimensional metaverse space is experienced by a user using a virtual reality headset and an augmented reality-enabled device.

[2189] (Claim 3)

[2190] The system of claim 1, wherein the server uses computer vision technology to analyze the photos or videos. [Explanation of symbols]

[2191] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. Photos or videos acquired from the user's device A server receives the photograph or video and analyzes the scenery, people, and movements of the photograph or video; A means for generating a three-dimensional metaverse space by a generation AI based on the analysis results; A means for storing the generated 3D metaverse space and providing it so that users can access it; A system including:

2. The system of claim 1 , wherein the generated three-dimensional metaverse space is experienced by a user using a VR headset and an AR-enabled device.

3. 2. The system of claim 1, wherein the server uses computer vision technology to analyze the photos or videos.

Images