System

Abstract

A system is provided.SOLUTION: A system including means for uploading selected image data by a terminal of a user, means for receiving the uploaded image data by a server and passing the image data to a generative artificial intelligence, means for analyzing the image data received by the generative artificial intelligence and converting the image data into data for a three dimensional virtual space, means for storing the converted three dimensional data in a virtual space platform by the server, and means for allowing the user to access the virtual space platform via the terminal and experience a reproduced memory.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Previously, photos and videos could only be saved in two-dimensional format, making it difficult to realistically relive memories and recollections. Furthermore, playing still images or videos made it impossible to recreate the emotions and movements of the time, preventing users from experiencing their memories deeply. The present invention aims to solve these problems by recreating memories in a three-dimensional virtual space based on the photos and videos saved by users, allowing them to experience them in a more realistic and immersive way. [Means for solving the problem]

[0005] The present invention provides a system including a means for uploading selected image data to a user's terminal, a means for a server to receive the uploaded image data and pass it to a generating artificial intelligence, a means for the generating artificial intelligence to analyze the received image data and convert it into data for a three-dimensional virtual space, a means for the server to store the converted three-dimensional data in a virtual space platform, and a means for the user to access the virtual space platform through their terminal and experience the reproduced memories. Furthermore, the system includes a means for notifying the user of the three-dimensional data stored in the virtual space platform and a means for reproducing the emotions and movements at the time the user selected based on the analyzed image data, thereby achieving a realistic reproduction of memories.

[0006] "User's device" refers to an electronic device used by a user, such as a computer, smartphone, or tablet.

[0007] "Image data" refers to visual information stored in digital format, such as photographs and videos.

[0008] "Means for uploading" refers to the functions and processes for sending image data from the user's terminal to the server.

[0009] "Server" means a central system that receives, stores, and processes data.

[0010] "Generative AI" refers to AI technology that analyzes image data and generates three-dimensional data.

[0011] "Analysis" refers to the process of extracting necessary information from input data and understanding and interpreting it.

[0012] "Data for three-dimensional virtual space" refers to 3D model data for visual and spatial representation in a virtual reality environment.

[0013] "Means for conversion" refers to the process or tool for converting image data into data for three-dimensional virtual space.

[0014] "Means for storing" refers to the functions and processes for recording the generated three-dimensional data on the virtual space platform.

[0015] A "virtual space platform" refers to a system or service that allows users to experience a virtual reality environment.

[0016] "Means of access" refers to the functions and processes that allow a user to connect to and use the virtual space platform via a terminal.

[0017] "Reproduce" means to return a past state or situation to its original form as closely as possible.

[0018] "Means for notifying" refers to the functions and processes for conveying information to the user about the stored three-dimensional data.

[0019] Reproducing "emotions and movements" refers to recreating the atmosphere and movements of people at that time in a virtual space based on image data. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] The present invention provides a system for recreating memories in a virtual space based on photos and videos saved by a user. The system includes a user terminal, a server, a generating artificial intelligence, and a virtual space platform.

[0042] First, the user's device has the function to upload selected image data (photos and videos). The user selects and uploads image data from their device through a dedicated application. An upload request is generated and the image data is sent to the server.

[0043] The server saves the received image data in storage. Once saving is complete, it passes the file path to the generating AI. The generating AI begins analyzing the image data and extracts the target scenery, people, movement patterns, etc. Based on this, data for the three-dimensional virtual space is generated.

[0044] The generated 3D data is returned to the server and stored in the virtual space platform. After the server confirms that the data has been saved, it sends a notification to the user. This notification includes the URL and login information for accessing the virtual space platform.

[0045] Users log in to the virtual space platform and use the URL to access the recreated space. Using a VR headset or PC, users can walk around the virtual space and interact with the scenery and people from their memories.

[0046] Specific examples

[0047] Example of recreating family trip memories

[0048] An example will be described in which a virtual space is recreated based on photos taken by a user on a summer vacation trip with his or her family.

[0049] 1. Upload:

[0050] A user launches the application on their smartphone, clicks the "Upload Photos" button in the application, selects a few photos from a family trip, and uploads them to the server using a dedicated function in the application.

[0051] 2. Data storage:

[0052] The server receives the uploaded photo and saves it in storage. Once the saving is complete, the file path is passed to the AI ​​generator.

[0053] 3. Image Analysis:

[0054] The artificial intelligence system receives the file path and begins analyzing the image. The system extracts images of family members and scenery from travel destinations (such as the ocean or mountains), and generates data for a three-dimensional virtual space based on this information.

[0055] 4. Data Retention and Notification:

[0056] The server receives the generated 3D data and uploads it to the virtual space platform for storage. After storage is complete, the server notifies the user of the completion of storage and provides access information.

[0057] 5. Virtual Experience:

[0058] The user logs into the virtual space platform using the provided access information. Putting on a VR headset, the user can relive memories of their family trip in the virtual space created. They can walk around, interact with the recreated scenery and people, and fully enjoy the experience.

[0059] In this way, this system allows users to recreate memories from their photos and videos in a realistic and immersive way. It can quickly respond to sudden changes in the environment and screen transitions, allowing users to relive their past memories without stress.

[0060] The processing flow will be explained below.

[0061] Step 1:

[0062] The user launches the app and selects photos and videos stored on their device. When the user clicks the "Upload" button, the selected image data is sent to the server as an upload request.

[0063] Step 2:

[0064] The device generates an upload request, attaches the selected image data, and sends it to the server. The data is transferred using a secure communication protocol (e.g., HTTPS).

[0065] Step 3:

[0066] The server receives the upload request and saves the image data in storage. The saved data is registered in the database along with the file path.

[0067] Step 4:

[0068] The server passes the saved file path to the generation artificial intelligence (AI) module, which then reads the image data based on this file path.

[0069] Step 5:

[0070] Generative AI analyzes image data to detect scenery, people, and movements in photos and videos. The analysis results are generated as three-dimensional data (e.g., 3D models, animations).

[0071] Step 6:

[0072] The generated 3D data is sent back from the AI ​​module to the server, which then receives the data and uploads it to the virtual space platform.

[0073] Step 7:

[0074] The virtual space platform stores the uploaded 3D data and generates a URL and login information that the user can access. The server receives the access information returned from the platform.

[0075] Step 8:

[0076] The server sends the user a URL and login information for accessing the virtual space platform via email or in-app notification.

[0077] Step 9:

[0078] The user receives the notification and accesses the specified URL, where they log in to the virtual space platform and enter the recreated space of their memories.

[0079] Step 10:

[0080] Users can use a VR headset or PC to walk around the virtual space and interact with the recreated scenery and people, allowing them to experience memories of that time in a realistic way.

[0081] Through the above processing steps, the user's image data is reproduced in a three-dimensional virtual space, allowing the user to re-experience memories in a realistic and immersive way.

[0082] Example 1

[0083] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0084] Conventional technologies for recreating memories based on photos and videos taken by users lack a sense of immersion, making it difficult to realistically recreate past experiences. Conventional systems simply display photos and videos on a flat screen, making it difficult to provide an experience that makes users feel as if they are actually there. This leads to a problem of a decrease in the excitement and realism felt when users relive past memories.

[0085] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0086] In this invention, the server includes a means for uploading image data selected by the user's information processing device, a means for the central processing unit to receive the uploaded image data and pass it to the generating AI, and a means for the generating AI to analyze the received image data and convert it into data for a three-dimensional virtual environment, thereby enabling the user to re-experience past memories in a more realistic and immersive virtual space.

[0087] A "user's information processing device" generally refers to a device that has computing resources for a user to operate, such as a smartphone or personal computer.

[0088] "Central Processing Unit" refers to a server or cloud service that has the computing resources to receive, store, and manage the analysis results of data.

[0089] "Generative AI" refers to an AI model that has the ability to analyze image data and generate three-dimensional virtual environment data based on the analysis results.

[0090] "Virtual environment platform" refers to the software and hardware infrastructure for storing generated three-dimensional virtual environment data and making it accessible to users.

[0091] "Image data" refers to visual information such as photos and videos uploaded by users.

[0092] "3D data" refers to 3D models and spatial information generated by generative artificial intelligence and usable within a virtual environment.

[0093] The term "uploading means" refers to a method or system for a user to operate an information processing device to transmit image data to a central processing unit.

[0094] The term "receiving means" refers to a method or system for the central processing unit to receive image data transmitted from the user's information processing device.

[0095] "Analysis means" refers to a method or system by which the generative artificial intelligence analyzes image data and converts it into three-dimensional data.

[0096] "Conversion means" refers to a method or system by which the generating artificial intelligence generates three-dimensional data based on image data.

[0097] "Storage means" refers to a method or system for recording the generated three-dimensional data in the virtual environment platform.

[0098] "Access means" refers to a method or system for a user to connect to the virtual environment platform via an information processing device and access a virtual environment generated based on the stored three-dimensional data.

[0099] This invention relates to a system that recreates memories in a virtual environment based on image data taken by the user. This system consists of a user's device, a server, a generative artificial intelligence (AI), and a virtual environment platform.

[0100] First, the user's information processing device (e.g., smartphone or personal computer) uploads image data (photos or videos). The user uses a dedicated application to select image data stored on the device and perform the upload operation. This operation causes the selected image data to be sent from the device to the server.

[0101] The server receives and saves the uploaded image data in its own storage. Once saved, the server passes the file path to a generative artificial intelligence (AI). The generative AI analyzes the image data using models such as OpenAI's DALL-E or GPT. This analysis extracts information such as people, scenery, and movement patterns from the image, and generates 3D virtual environment data.

[0102] The generated 3D data is returned to the server and saved in the virtual environment platform (e.g., Unity or Unreal Engine). After saving is complete, the server sends a notification to the user. This notification includes the URL and login information for accessing the virtual environment platform.

[0103] The user logs into the virtual environment platform using the provided information. The user can access the virtual space using a VR headset or a PC and experience the recreated memories. Within this virtual environment, the user can walk around and interact with the recreated scenery and people.

[0104] Specific examples

[0105] Example of recreating family trip memories

[0106] 1. User uploads image data:

[0107] Users select photos taken during their summer vacation with their family on their smartphone and upload them using a dedicated application.

[0108] 2. Data storage by the server:

[0109] The server receives the uploaded photos and saves them in the specified storage path. After saving, the file path is passed to the generation AI.

[0110] 3. Analysis of image data and generation of 3D data:

[0111] The generative artificial intelligence analyzes and extracts images of family members and travel destinations from the photos, and generates three-dimensional virtual environment data based on this.

[0112] 4. 3D data storage and notification:

[0113] The server stores the three-dimensional data in the virtual environment platform, and after the storage is complete, notifies the user of the access information.

[0114] 5. Virtual Experience:

[0115] The user logs into the virtual environment platform using the provided information and uses a VR headset to experience the recreated memories.

[0116] Prompt Sentence Examples

[0117] "Recreate a beach memory using photos from your family summer vacation. Choose photos that show the whole family, including walking along the shore and playing in the ocean."

[0118] In this way, users can recreate their past memories in a virtual space and have a realistic experience, allowing them to vividly recall past experiences.

[0119] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0120] Step 1:

[0121] The user selects the image data and starts uploading.

[0122] Input: Image data (photos and videos) selected by the user using an application on a smartphone or PC.

[0123] How it works: The user launches the dedicated application, clicks the "Upload Photos" button, and selects the image data they want to upload from their device.

[0124] Output: The selected image data is saved to the device as an upload request.

[0125] Step 2:

[0126] The terminal transmits the image data to the server.

[0127] Input: Image data selected and saved by the user.

[0128] How it works: The device sends an upload request to the server, which uses its internet connection to split the image data into packets and begin sending them to the server.

[0129] Output: Packets of image data received by the server.

[0130] Step 3:

[0131] The server receives and stores the image data.

[0132] Input: Packets of image data sent from the device.

[0133] Operation: The server receives packets sequentially, integrates them, and saves them as complete image data in storage. After saving is complete, it records the path of the saved file.

[0134] Output: The saved image data and its file path in storage.

[0135] Step 4:

[0136] The server passes the file path to the generation AI.

[0137] Input: The file path of the saved image data.

[0138] Operation: The server generates a request to send the file path of the stored image data to the generating artificial intelligence and sends it over the network.

[0139] Output: The file path received by the generating AI.

[0140] Step 5:

[0141] Generative artificial intelligence analyzes image data and generates three-dimensional data.

[0142] Input: The file path of the received image data.

[0143] How it works: Generative AI (e.g., OpenAI's DALL-E or GPT model) analyzes image data based on file paths. The system extracts patterns of people, scenery, and movement within the image and generates 3D virtual environment data based on this data.

[0144] Output: Generated 3D virtual environment data.

[0145] Step 6:

[0146] The generated three-dimensional data is received by the server and stored in the virtual environment platform.

[0147] Input: 3D virtual environment data returned from a generative AI model.

[0148] How it works: The server receives the 3D data and uploads it to the virtual environment platform (e.g. Unity or Unreal Engine). It verifies the integrity of the data and checks that it has been saved correctly.

[0149] Output: Three-dimensional virtual environment data stored in the virtual environment platform.

[0150] Step 7:

[0151] The server sends a notification to the user.

[0152] Input: Three-dimensional virtual environment data stored in the virtual environment platform.

[0153] What it does: The server notifies the user that the save is complete. The notification includes a URL and login information for accessing the virtual environment platform.

[0154] Output: The notification the user receives, containing access information.

[0155] Step 8:

[0156] A user accesses the virtual environment platform.

[0157] Input: The URL and login information contained in the notification from the server.

[0158] Operation: The user accesses the virtual environment platform based on the notification and enters their login information. Once logged in, they can access the recreated space of their memories.

[0159] Output: Recreated memory data within the virtual environment accessed by the user.

[0160] Step 9:

[0161] Users can experience a memorable virtual space using VR.

[0162] Input: Recreated memory data in a virtual environment.

[0163] How it works: Using a VR headset or PC, users can walk around the virtual space and interact with the recreated scenery and people. Through the experience in the virtual space, they can relive past memories.

[0164] Output: An immersive virtual memory space experienced by the user.

[0165] Through the above processing steps, the user can re-experience the virtual space with a sense of realism based on the image data he or she has taken.

[0166] (Application example 1)

[0167] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0168] Simply storing a user's photos and videos visually is difficult to fully recreate the sense of presence and depth of memories. Furthermore, conventional systems make it difficult for users to move and interact within the virtual space, limiting the user experience. Thus, technology is needed to recreate real-world experiences in virtual spaces and provide a deeper sense of immersion.

[0169] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0170] In this invention, the server includes means for uploading media data selected by the user's terminal, means for receiving the uploaded media data and passing it to the generating AI, means for the generating AI to analyze the received media data and convert it into data for a three-dimensional virtual space, means for saving the converted three-dimensional data in the virtual space platform, means for the user to access the virtual space platform through the terminal and experience the recreated memories, means for the user to move and interact within the generated three-dimensional space, means for notifying the user of the three-dimensional data saved in the virtual space platform, and means for recreating the emotions and movements at the time selected by the user based on the analyzed media data, thereby enabling the user to have a more immersive virtual space experience.

[0171] "User terminal" refers to an electronic device used by a user, including a smartphone, tablet, or PC.

[0172] "Media Data" means data in digital form that contains visual or audio information, such as photographs or videos.

[0173] "Upload" is the process of sending data from a user's device to a server.

[0174] A "server" is a computer system installed to provide services over a network, and stores and manages data.

[0175] "Generative AI" refers to algorithms and models that analyze media data uploaded by users and generate new data.

[0176] "Analysis" is the process of examining received data in detail to identify and extract specific information.

[0177] "Data for three-dimensional virtual space" is digital data that has been converted into a format that can be visually displayed in three-dimensional space.

[0178] A "virtual space platform" is a software and hardware system that provides virtual reality (VR) and augmented reality (AR) environments.

[0179] "Notifying" is the process by which a system communicates some information to a user.

[0180] "Movement and interaction" refers to the user's actions of walking around the virtual space and interacting with other objects and characters.

[0181] "Reproducing emotions and movements" is the process of expressing a user's emotional state and physical movements at a particular point in time in a virtual space.

[0182] The present invention relates to a system for recreating memories in a virtual space based on media data such as photos and videos saved by a user. The system includes a user terminal, a server, a generating artificial intelligence, and a virtual space platform.

[0183] First, a dedicated application is installed on the user's device. The user uses this application to select and upload photos and videos from their device. An upload request is generated and the selected media data is sent to the server.

[0184] The server saves the uploaded media data in storage. Once saving is complete, it passes the file path to the generation AI. The generation AI receives this file path and begins analyzing the image data. This analysis uses computer vision technology (e.g., OpenCV) and machine learning models (e.g., TensorFlow). As a result of the analysis, the target person or object, background scenery, etc. are extracted.

[0185] The generative AI generates three-dimensional virtual space data based on the extracted information. This three-dimensional data is returned to the server and stored on the virtual space platform. After the server confirms that the data has been saved, it sends a notification to the user. This notification includes a URL and login information for accessing the virtual space platform.

[0186] The user logs into the virtual space platform using the provided access information. Using smart glasses or a head-mounted display, the user can access the generated virtual space. The user can walk around and interact within the virtual space. This allows the user to enjoy a realistic and immersive experience as a recreated memory.

[0187] For example, consider a specific scenario in which a virtual space is recreated based on photos taken on a family trip. The user opens a smartphone app, selects "family trip photos," and uploads them. The server receives the photos, performs image analysis, and generates a three-dimensional space. The user then logs in to the virtual space using the provided access information and puts on a VR headset, allowing them to re-experience the family trip in the created virtual space.

[0188] An example of a prompt to give to a generative AI model is:

[0189] "Create a virtual space that recreates family vacation memories from photos and videos uploaded by users. Identify specific landscapes, extract people's figures, and generate 3D models based on that data."

[0190] This system allows users to enjoy a more immersive virtual space experience as a recreated memory.

[0191] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0192] Step 1:

[0193] The user launches the smartphone application, selects saved photos and videos, generates an upload request for the selected media data, and sends the data to the server.

[0194] Input: Photos and videos stored on the user's device

[0195] Output: Upload request sent to the server

[0196] Specific operation: When a user selects a photo or video within the app and presses the "Upload" button, the media data is sent to the server.

[0197] Step 2:

[0198] The server receives an upload request from the user and saves the media data in storage. Once the saving is complete, the server passes the file path to the generation AI.

[0199] Input: User-uploaded photo and video data

[0200] Output: The file path saved in the storage

[0201] Specific operation: The server saves the media data in the file system, obtains the file path of the saved location, and provides it to the generating artificial intelligence.

[0202] Step 3:

[0203] The generative AI receives the file path and begins analyzing the image data. It uses libraries such as OpenCV and TensorFlow to extract the target person, object, background scenery, etc. As a result of the analysis, it generates data for a 3D virtual space.

[0204] Input: File path of media data stored in storage

[0205] Output: Data for 3D virtual space

[0206] Specific operation: The generative AI loads the image data, runs an analysis algorithm to extract the necessary information, and generates a three-dimensional model based on that information.

[0207] Step 4:

[0208] The server receives the 3D data generated by the AI ​​generator and stores it on the virtual space platform. After the data has been stored, the server sends a notification to the user. The notification includes access information to the virtual space platform (URL, login information, etc.).

[0209] Input: Data for 3D virtual space

[0210] Output: User notification

[0211] Specific operation: The server uploads the 3D data to the virtual space platform and sends a save completion notification to the user.

[0212] Step 5:

[0213] The user logs into the virtual space platform using the access information provided in the notification. The user accesses the virtual space by wearing a device such as smart glasses or a head-mounted display, and experiences the recreated memories. The user can walk around and interact with the virtual space.

[0214] Input: Access information (URL, login information), user interface device (smart glasses, head-mounted display, etc.)

[0215] Output: User experience in virtual space

[0216] Specific operations: The user logs into the virtual space platform using the provided URL and uses the VR device to explore the virtual space and interact with the recreated pictures and people.

[0217] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0218] The present invention is a system for recreating memories in a virtual space based on photos and videos saved by the user, and recognizing the user's emotions to enrich the recreating process. This system includes a user terminal, a server, a generative artificial intelligence, an emotion engine, and a virtual space platform.

[0219] First, the user's device has the function to upload selected image data (photos and videos). The user selects and uploads image data from their device through a dedicated application. An upload request is generated and the image data is sent to the server.

[0220] The server saves the received image data in storage. Once saving is complete, it passes the file path to the generating AI. The generating AI begins analyzing the image data and extracts the target scenery, people, movement patterns, etc. Based on this, data for the three-dimensional virtual space is generated.

[0221] Next, the emotion engine analyzes the image data and recognizes the user's emotions. Based on this recognized emotion information, the generative AI makes further fine adjustments. Specifically, it adjusts the environmental settings (e.g., lighting and music) and effects (visual effects such as warmth and coolness) in the virtual space based on the emotion information.

[0222] The generated 3D data and the data reflecting the emotion information are returned to the server and saved on the virtual space platform. After the server confirms that saving is complete, it sends a notification to the user. This notification includes the URL and login information for accessing the virtual space platform.

[0223] Users log in to the virtual space platform and access the recreated space using the URL they receive. Using a VR headset or PC, users can walk around the virtual space and interact with the scenery and people from their memories. Furthermore, the recreated space has environmental settings and effects that are adjusted based on the user's emotions, making for a more realistic and emotional experience.

[0224] Specific examples

[0225] Example of recreating family trip memories

[0226] We will explain an example of recreating a virtual space based on photos taken by a user on a summer vacation with their family, and also reflecting the emotions they felt at the time.

[0227] 1. Upload:

[0228] A user launches the application on their smartphone, clicks the "Upload Photos" button in the application, selects a few photos from a family trip, and uploads them to the server using a dedicated function in the application.

[0229] 2. Data storage:

[0230] The server receives the uploaded photo and saves it in storage. Once the saving is complete, the file path is passed to the AI ​​generator.

[0231] 3. Image and Sentiment Analysis:

[0232] The artificial intelligence system receives the file path and begins analyzing the image. The system extracts images of family members and scenery from travel destinations (such as the ocean or mountains), and generates data for a three-dimensional virtual space based on this information.

[0233] At the same time, the emotion engine analyzes the image data and recognizes the user's emotions (e.g., joy, surprise, excitement).

[0234] 4. Environment and Effects Settings:

[0235] Based on the emotional information recognized by the emotion engine, the generative AI adjusts the environmental settings (e.g., music and lighting) and effects (e.g., specific visual effects) in the virtual space, thereby reinforcing the emotions felt by the user at the time.

[0236] 5. Data Retention and Notification:

[0237] The server receives the generated 3D data and the data reflecting the emotional information, uploads it to the virtual space platform, and saves it. After saving is complete, the server notifies the user of the completion of saving and provides access information.

[0238] 6. Virtual Experience:

[0239] Users log in to the virtual space platform using the provided access information. They then put on a VR headset and can relive their family trip memories in the virtual space created. They can walk around, interact with the recreated scenery and people, and enjoy the environmental settings and effects that recreate the emotions of the time.

[0240] In this way, by recognizing the user's emotions and recreating memories based on those emotions, it is possible to provide a more realistic and moving experience.

[0241] The processing flow will be explained below.

[0242] Step 1:

[0243] The user launches the app and selects photos and videos stored on their device. When the user clicks the "Upload" button, the selected image data is sent to the server as an upload request.

[0244] Step 2:

[0245] The device generates an upload request, attaches the selected image data, and sends it to the server. The data is transferred using a secure communication protocol (e.g., HTTPS).

[0246] Step 3:

[0247] The server receives the upload request and saves the image data in storage. The saved data is registered in the database along with the file path.

[0248] Step 4:

[0249] The server passes the saved file path to the generation artificial intelligence (AI) module, which then reads the image data based on this file path.

[0250] Step 5:

[0251] Generative AI analyzes image data and detects scenery, people, and movements in photos and videos. Based on this, data for three-dimensional virtual space (3D models and animations) is generated.

[0252] Step 6:

[0253] The server passes the image data to the emotion engine, which then recognizes the user's emotion (e.g., joy, excitement, surprise) from the image data.

[0254] Step 7:

[0255] The emotion engine provides the recognized emotion information to the generation AI, which then adjusts the environmental settings (e.g., lighting, music) and visual effects in the virtual space based on this emotion information.

[0256] Step 8:

[0257] The artificial intelligence generates and adjusts the 3D data and environment setting data, which are then returned to the server, which then uploads and stores them on the virtual space platform.

[0258] Step 9:

[0259] The virtual space platform stores the uploaded data and generates a URL and login information that the user can access. The server receives the access information returned by the platform.

[0260] Step 10:

[0261] The server sends the user a URL and login information for accessing the virtual space platform via email or in-app notification.

[0262] Step 11:

[0263] The user receives the notification and accesses the specified URL, where they log in to the virtual space platform and enter the recreated space of their memories.

[0264] Step 12:

[0265] Users can walk around the virtual space using a VR headset or PC, interacting with the recreated scenery and people. Environmental settings and visual effects based on the user's emotional information further enrich the experience.

[0266] Through the above processing steps, a realistic and moving memory can be reproduced in a three-dimensional virtual space using the user's image data.

[0267] Example 2

[0268] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0269] Conventional technologies have limited means for users to recreate their individual memories in a virtual space based on photos and videos they have taken. It has also been difficult to provide a more realistic and moving experience by reflecting the user's emotions. Therefore, it is necessary to recognize the user's emotions and provide appropriate environmental settings and effects to enrich the recreation of memories in a virtual space.

[0270] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0271] In this invention, the server includes means for receiving uploaded image data and saving it in storage, means for passing the path of the saved image file to the generating AI, and means for saving the generated three-dimensional data and data reflecting emotional information in the virtual space platform. This makes it possible to generate and save a three-dimensional virtual space that takes emotional information into consideration based on image data taken by the user, and to provide the user with the experience of recreating moving memories.

[0272] "User terminal" refers to any electronic device that a user operates to send and receive data.

[0273] A "server" is a computer system that receives, stores, processes, and provides data over a network.

[0274] "Image data" refers to digital files that contain visual information, such as photographs and videos.

[0275] "Storage" refers to a physical or virtual medium for storing digital data.

[0276] "Generative AI" refers to artificial intelligence techniques for analyzing and generating data.

[0277] "Data for three-dimensional virtual space" refers to all digital information used to represent virtual space using three-dimensional computer graphics.

[0278] An "emotion engine" refers to technology that includes an algorithm for analyzing a user's emotions and generating an appropriate response based on that.

[0279] "Virtual space platform" refers to an integrated system of software and hardware that allows users to access and experience virtual spaces.

[0280] "Notification" refers to a message or alert that the system uses to communicate information to the user.

[0281] MODE FOR CARRYING OUT THE INVENTION

[0282] This invention is a system that recreates memories in a virtual space based on photos and videos saved by the user, and further enriches the recreation by recognizing the user's emotions. This system consists of a user terminal, a server, a generative artificial intelligence, an emotion engine, and a virtual space platform. The detailed processing of each hardware and software is described below.

[0283] Hardware and Software

[0284] User device: Refers to electronic devices such as smartphones, PCs, tablets, etc. Users select and upload image data using a dedicated application.

[0285] Server: A computer system that receives, stores, processes, and provides data. The server is equipped with storage and stores image data and generated 3D data.

[0286] Generative AI: AI technology that analyzes image data and generates 3D data. It extracts specific landscapes, people, and movement patterns to generate data for 3D virtual spaces.

[0287] Emotion engine: A technology that analyzes image data to recognize the user's emotions and generates environment settings and effects that reflect those emotions.

[0288] Virtual Space Platform: A system that allows users to access virtual spaces and relive memories. It can be accessed using a VR headset or a PC.

[0289] Program processing

[0290] Processing on the user's device

[0291] A user starts a dedicated application and selects and uploads photos and videos from their device. At this time, the device generates an upload request and sends the selected image data to the server. For example, a user selects and uploads several photos taken during a summer vacation with their family.

[0292] Processing on the server

[0293] The server receives the image data sent from the device and saves it in storage. Once the saving is complete, the server passes the file path to the AI ​​generator. For example, the server provides the AI ​​generator with the file path of a set of saved family trip photos.

[0294] Processing with generative artificial intelligence

[0295] The artificial intelligence begins analyzing the image data based on the file path received from the server. This allows it to extract the target scenery, people, and movement patterns, and generate data for a 3D virtual space. In this case, it recreates the beach scenery and family figures in 3D from photos of a family summer vacation.

[0296] Processing with the Emotion Engine

[0297] The emotion engine analyzes the image data and recognizes the user's emotions (e.g., joy, surprise, excitement). Based on this, the generative AI adjusts the environmental settings and effects of the virtual space. For example, if the user's emotion is recognized as "joy," the brightness and music in the virtual space will be adjusted.

[0298] Processing on a virtual space platform

[0299] The generated 3D data and the data reflecting the emotional information are sent back to the server and saved on the virtual space platform. Once saved, the server sends a notification to the user. The user can access the platform using the URL and login information in the notification and experience the memories recreated in the virtual space using a VR headset or PC.

[0300] Specific examples

[0301] Example of recreating family trip memories

[0302] An example of recreating a virtual space based on photos taken by a user on a summer vacation with their family would include the following steps:

[0303] 1. Upload

[0304] The user launches a dedicated app on their smartphone, selects a few photos from a family trip, and clicks the upload button.

[0305] 2. Data storage

[0306] The server receives the image data and stores it in storage.

[0307] 3. Image and Sentiment Analysis

[0308] Generative AI analyzes photos and an emotion engine recognizes emotions.

[0309] 4. Environment and Effects Settings

[0310] Adjusting environmental settings and effects in a virtual space based on emotional information.

[0311] 5. Data Retention and Notification

[0312] The server stores the generated data in the virtual space platform and notifies the user.

[0313] 6. Virtual Experience

[0314] Users use a VR headset to experience a virtual space that recreates memories of a family trip.

[0315] Prompt Sentence Examples

[0316] "Project: Memories of a summer family trip. Upload family photos on the screen. Then, we analyze emotions and add appropriate effects. For example, we create an environment that evokes feelings of joy or surprise."

[0317] This system allows users to recreate and experience memories more realistically based on the emotions they felt.

[0318] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0319] Step 1:

[0320] The user selects and uploads image data from the device using a dedicated application. Specifically, the user launches the smartphone app and clicks the "Upload Photos" button. They then select multiple photos from a family trip and press the upload button. The device generates an upload request based on this and sends the image data to the server.

[0321] Input: Image data selected by the user (photos, videos)

[0322] Output: Upload request to server

[0323] Step 2:

[0324] The server receives the image data sent from the terminal and stores it in storage. Specifically, the server stores the received image data file in a specified directory and registers the file path in the management system.

[0325] Input: Image data sent from the device

[0326] Output: Image file saved in storage, file path obtained

[0327] Step 3:

[0328] The server passes the file path of the stored image data to the generation AI. Specifically, the server provides the path of the image file as an input parameter of the generation AI and requests analysis.

[0329] Input: File path of saved image data

[0330] Output: Analysis request to the generative AI

[0331] Step 4:

[0332] Generative AI analyzes image data and extracts scenery, people, movement patterns, etc. For example, it can recognize scenery such as the sea and mountains, as well as family members' faces and movements from photos of a family trip, and use this information to generate data for a three-dimensional virtual space.

[0333] Input: Image data file path

[0334] Output: Data for 3D virtual space

[0335] Step 5:

[0336] The emotion engine analyzes image data and recognizes the user's emotions. Specifically, the emotion engine analyzes facial expressions and situations obtained from the photos to detect emotions such as joy and surprise. This emotional information is provided to the generative AI.

[0337] Input: Image data

[0338] Output: Recognized emotion information

[0339] Step 6:

[0340] The generative AI adjusts the environmental settings and effects in the virtual space based on the emotional information obtained from the emotion engine. Specifically, it selects the brightness and color of the lighting, background music, etc. based on the emotional information. For example, if joy is recognized, bright lighting and upbeat music will be set.

[0341] Input: Recognized emotion information

[0342] Output: Adjusted virtual space settings and effects

[0343] Step 7:

[0344] The server stores the data generated by the generative AI and emotion engine on the virtual space platform. Specifically, the server uploads the generated three-dimensional data and emotion information to the virtual space platform and stores them as data.

[0345] Input: 3D data, emotional information

[0346] Output: Data stored in the virtual space platform

[0347] Step 8:

[0348] The server notifies the user that the save has been completed. Specifically, the server sends a notification to the user's device that includes an access URL and login information.

[0349] Input: Completion information saved in the virtual space platform

[0350] Output: Notification to user (access URL, login information)

[0351] Step 9:

[0352] Based on the information provided, users can log in to the virtual space platform and experience the recreated memories. Specifically, users access the virtual space using a VR headset or PC and interact with the recreated scenery and people. They experience environmental settings and effects based on emotional information.

[0353] Input: Access URL, login information

[0354] Output: Experience in virtual space (recreated memory interaction)

[0355] (Application example 2)

[0356] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0357] Conventional virtual space reproduction systems based on photographs and videos have the problem that they are unable to create environmental settings and effects that take into account the user's emotions, resulting in a lack of realism in the experience. The present invention aims to solve this problem and provide a more emotionally rich and realistic virtual space experience.

[0358] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading image data selected by the user's terminal, means for the server to receive the uploaded image data and pass it to the generation AI, means for the generation AI to analyze the received image data and convert it into data for a three-dimensional virtual space, means for the emotion engine to analyze the image data and recognize the user's emotional information, means for the generation AI to adjust the environmental settings and effects of the virtual space based on the emotional information, means for the server to save the converted three-dimensional data in the virtual space platform, and means for the user to access the virtual space platform via their terminal and experience recreated memories. This enables a rich and realistic virtual space experience that reflects the user's emotions.

[0359] "User terminal" refers to a device used by a user to select and upload image data, and includes smartphones, tablets, PCs, etc.

[0360] "Image data" refers to data of photos and videos saved by the user, and is the basis for recreating the virtual space.

[0361] A "server" is a computer system that receives and stores uploaded image data, and also exchanges data with the generating artificial intelligence and virtual space platform.

[0362] "Generative AI" is an AI technology that analyzes image data and generates data for three-dimensional virtual space.

[0363] The "emotion engine" is a technology that recognizes the user's emotional information from analyzed image data and adjusts the environmental settings and effects of the virtual space based on that information.

[0364] A "virtual space platform" is a system that stores generated three-dimensional data and provides a virtual space for users to access and experience.

[0365] "Three-dimensional data" is data in a three-dimensional virtual space generated based on image data, and is data that represents objects and environments within the virtual space.

[0366] "Environmental settings" are settings that adjust environmental elements such as lighting and music within the virtual space.

[0367] "Effects" are visual and tactile effects that enrich the user's experience within the virtual space.

[0368]

[0369] The present invention is a system that recreates memories in a virtual space based on photos and videos saved by the user, and further enriches the recreation by recognizing the user's emotions. This system includes a user terminal, a server, a generative artificial intelligence, an emotion engine, and a virtual space platform.

[0370] First, the user's device has the function to upload selected image data (photos and videos). The user selects and uploads image data from their device through a dedicated application. An upload request is generated and the image data is sent to the server.

[0371] The server saves the received image data in storage. Once saving is complete, it passes the file path to the generating AI. The generating AI begins analyzing the image data and extracts the target scenery, people, movement patterns, etc. Based on this, data for the three-dimensional virtual space is generated.

[0372] Next, the emotion engine analyzes the image data and recognizes the user's emotions. Based on this recognized emotion information, the generative AI makes further fine adjustments. Specifically, it adjusts the environmental settings (e.g., lighting and music) and effects (visual effects such as warmth and coolness) in the virtual space based on the emotion information.

[0373] The generated 3D data and the data reflecting the emotional information are returned to the server and saved on the virtual space platform. After the server confirms that saving is complete, it sends a notification to the user. This notification includes the URL and login information for accessing the virtual space platform.

[0374] Users log in to the virtual space platform and access the recreated space using the URL they receive. Using a VR headset or PC, users can walk around the virtual space and interact with the scenery and people from their memories. Furthermore, the recreated space has environmental settings and effects that are adjusted based on the user's emotions, making for a more realistic and emotional experience.

[0375] Specific examples

[0376] Example of recreating family trip memories

[0377] 1. Upload:

[0378] A user launches the application on their smartphone, clicks the "Upload Photos" button in the application, selects a few photos from a family trip, and uploads them to the server using a dedicated function in the application.

[0379] 2. Data storage:

[0380] The server receives the uploaded photo and saves it in storage. Once the saving is complete, the file path is passed to the AI ​​generator.

[0381] 3. Image and Sentiment Analysis:

[0382] The generation AI receives the file path and begins image analysis. The system extracts images of family members and travel destinations (such as the ocean or mountains) and generates data for a 3D virtual space based on this. At the same time, the emotion engine analyzes the image data and recognizes the user's emotions (e.g., joy, surprise, excitement).

[0383] 4. Environment and Effects Settings:

[0384] Based on the emotional information recognized by the emotion engine, the generative AI adjusts the environmental settings (e.g., music and lighting) and effects (e.g., specific visual effects) in the virtual space, thereby reinforcing the emotions felt by the user at the time.

[0385] 5. Data Retention and Notification:

[0386] The server receives the generated 3D data and the data reflecting the emotional information, uploads it to the virtual space platform, and saves it. After saving is complete, the server notifies the user of the completion of saving and provides access information.

[0387] 6. Virtual Experience:

[0388] Users log in to the virtual space platform using the provided access information. They then put on a VR headset and can relive their family trip memories in the virtual space created. They can walk around, interact with the recreated scenery and people, and enjoy the environmental settings and effects that recreate the emotions of the time.

[0389] Prompt Sentence Examples

[0390] "I uploaded three photos and one video of my family at a carnival. Use these to create a virtual space filled with fun and excitement. Add colorful lights and lively music."

[0391] This system enables a rich and realistic virtual space experience that reflects the user's emotions.

[0392] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0393] Step 1:

[0394] The user launches the application on their smartphone and clicks the dedicated "Photo Upload" button. The user selects several photos and videos from a family trip from within the application and generates an upload request. At this time, the selected image data is imported into the application.

[0395] Step 2:

[0396] The user's device sends the generated upload request to the server. The server receives this request and saves the image data to storage. Once saving is complete, the file path is recorded. The image data received as input is saved and a file path is generated as output.

[0397] Step 3:

[0398] The server passes the saved file path to the generation AI. The generation AI receives the file path and begins analyzing the image data. It extracts the target scenery, people, and movement patterns from the image data, and generates data for a three-dimensional virtual space based on that. It receives image data as input and generates three-dimensional data as output.

[0399] Step 4:

[0400] The emotion engine analyzes the image data received from the generation AI and recognizes the user's emotional information (e.g., enjoyment, surprise, excitement). Based on the recognized emotional information, the generation AI adjusts the environmental settings (e.g., lighting and music) and effects (visual effects such as warmth and coolness) of the virtual space. It receives the user's emotional information as input and generates adjusted three-dimensional data as output.

[0401] Step 5:

[0402] The generated 3D data and the data reflecting the emotion information are returned to the server. The server uploads and saves this data to the virtual space platform. After confirming that saving is complete, the server sends a notification to the user. This notification includes the URL and login information for accessing the virtual space platform. It receives 3D data as input and generates a user notification as output.

[0403] Step 6:

[0404] The user logs into the virtual space platform using the notified URL and login information. Using a VR headset or PC, the user relives the memories of their family trip in the created virtual space. The user can walk around the virtual space and interact with the recreated scenery and people. The recreated space has environmental settings and effects adjusted based on emotions, making for a more realistic and moving experience. The notified information is used as input, and access to and experience of the virtual space is obtained as output.

[0405] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0406] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0407] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0408] [Second embodiment]

[0409] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0410] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0411] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0412] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0413] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0414] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0415] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0416] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0417] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0418] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0419] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0420] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0421] The present invention provides a system for recreating memories in a virtual space based on photos and videos saved by a user. The system includes a user terminal, a server, a generating artificial intelligence, and a virtual space platform.

[0422] First, the user's device has the function to upload selected image data (photos and videos). The user selects and uploads image data from their device through a dedicated application. An upload request is generated and the image data is sent to the server.

[0423] The server saves the received image data in storage. Once saving is complete, it passes the file path to the generating AI. The generating AI begins analyzing the image data and extracts the target scenery, people, movement patterns, etc. Based on this, data for the three-dimensional virtual space is generated.

[0424] The generated 3D data is returned to the server and stored in the virtual space platform. After the server confirms that the data has been saved, it sends a notification to the user. This notification includes the URL and login information for accessing the virtual space platform.

[0425] Users log in to the virtual space platform and use the URL to access the recreated space. Using a VR headset or PC, users can walk around the virtual space and interact with the scenery and people from their memories.

[0426] Specific examples

[0427] Example of recreating family trip memories

[0428] An example will be described in which a virtual space is recreated based on photos taken by a user on a summer vacation trip with his or her family.

[0429] 1. Upload:

[0430] A user launches the application on their smartphone, clicks the "Upload Photos" button in the application, selects a few photos from a family trip, and uploads them to the server using a dedicated function in the application.

[0431] 2. Data storage:

[0432] The server receives the uploaded photo and saves it in storage. Once the saving is complete, the file path is passed to the AI ​​generator.

[0433] 3. Image Analysis:

[0434] The artificial intelligence system receives the file path and begins analyzing the image. The system extracts images of family members and scenery from travel destinations (such as the ocean or mountains), and generates data for a three-dimensional virtual space based on this information.

[0435] 4. Data Retention and Notification:

[0436] The server receives the generated 3D data and uploads it to the virtual space platform for storage. After storage is complete, the server notifies the user of the completion of storage and provides access information.

[0437] 5. Virtual Experience:

[0438] The user logs into the virtual space platform using the provided access information. Putting on a VR headset, the user can relive memories of their family trip in the virtual space created. They can walk around, interact with the recreated scenery and people, and fully enjoy the experience.

[0439] In this way, this system allows users to recreate memories from their photos and videos in a realistic and immersive way. It can quickly respond to sudden changes in the environment and screen transitions, allowing users to relive their past memories without stress.

[0440] The processing flow will be explained below.

[0441] Step 1:

[0442] The user launches the app and selects photos and videos stored on their device. When the user clicks the "Upload" button, the selected image data is sent to the server as an upload request.

[0443] Step 2:

[0444] The device generates an upload request, attaches the selected image data, and sends it to the server. The data is transferred using a secure communication protocol (e.g., HTTPS).

[0445] Step 3:

[0446] The server receives the upload request and saves the image data in storage. The saved data is registered in the database along with the file path.

[0447] Step 4:

[0448] The server passes the saved file path to the generation artificial intelligence (AI) module, which then reads the image data based on this file path.

[0449] Step 5:

[0450] Generative AI analyzes image data to detect scenery, people, and movements in photos and videos. The analysis results are generated as three-dimensional data (e.g., 3D models, animations).

[0451] Step 6:

[0452] The generated 3D data is sent back from the AI ​​module to the server, which then receives the data and uploads it to the virtual space platform.

[0453] Step 7:

[0454] The virtual space platform stores the uploaded 3D data and generates a URL and login information that the user can access. The server receives the access information returned from the platform.

[0455] Step 8:

[0456] The server sends the user a URL and login information for accessing the virtual space platform via email or in-app notification.

[0457] Step 9:

[0458] The user receives the notification and accesses the specified URL, where they log in to the virtual space platform and enter the recreated space of their memories.

[0459] Step 10:

[0460] Users can use a VR headset or PC to walk around the virtual space and interact with the recreated scenery and people, allowing them to experience memories of that time in a realistic way.

[0461] Through the above processing steps, the user's image data is reproduced in a three-dimensional virtual space, allowing the user to re-experience memories in a realistic and immersive way.

[0462] Example 1

[0463] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0464] Conventional technologies for recreating memories based on photos and videos taken by users lack a sense of immersion, making it difficult to realistically recreate past experiences. Conventional systems simply display photos and videos on a flat screen, making it difficult to provide an experience that makes users feel as if they are actually there. This leads to a problem of a decrease in the excitement and realism felt when users relive past memories.

[0465] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0466] In this invention, the server includes a means for uploading image data selected by the user's information processing device, a means for the central processing unit to receive the uploaded image data and pass it to the generating AI, and a means for the generating AI to analyze the received image data and convert it into data for a three-dimensional virtual environment, thereby enabling the user to re-experience past memories in a more realistic and immersive virtual space.

[0467] A "user's information processing device" generally refers to a device that has computing resources for a user to operate, such as a smartphone or personal computer.

[0468] "Central Processing Unit" refers to a server or cloud service that has the computing resources to receive, store, and manage the analysis results of data.

[0469] "Generative AI" refers to an AI model that has the ability to analyze image data and generate three-dimensional virtual environment data based on the analysis results.

[0470] "Virtual environment platform" refers to the software and hardware infrastructure for storing generated three-dimensional virtual environment data and making it accessible to users.

[0471] "Image data" refers to visual information such as photos and videos uploaded by users.

[0472] "3D data" refers to 3D models and spatial information generated by generative artificial intelligence and usable within a virtual environment.

[0473] The term "uploading means" refers to a method or system for a user to operate an information processing device to transmit image data to a central processing unit.

[0474] The term "receiving means" refers to a method or system for the central processing unit to receive image data transmitted from the user's information processing device.

[0475] "Analysis means" refers to a method or system by which the generative artificial intelligence analyzes image data and converts it into three-dimensional data.

[0476] "Conversion means" refers to a method or system by which the generating artificial intelligence generates three-dimensional data based on image data.

[0477] "Storage means" refers to a method or system for recording the generated three-dimensional data in the virtual environment platform.

[0478] "Access means" refers to a method or system for a user to connect to the virtual environment platform via an information processing device and access a virtual environment generated based on the stored three-dimensional data.

[0479] This invention relates to a system that recreates memories in a virtual environment based on image data taken by the user. This system consists of a user's device, a server, a generative artificial intelligence (AI), and a virtual environment platform.

[0480] First, the user's information processing device (e.g., smartphone or personal computer) uploads image data (photos or videos). The user uses a dedicated application to select image data stored on the device and perform the upload operation. This operation causes the selected image data to be sent from the device to the server.

[0481] The server receives and saves the uploaded image data in its own storage. Once saved, the server passes the file path to a generative artificial intelligence (AI). The generative AI analyzes the image data using models such as OpenAI's DALL-E or GPT. This analysis extracts information such as people, scenery, and movement patterns from the image, and generates 3D virtual environment data.

[0482] The generated 3D data is returned to the server and saved in the virtual environment platform (e.g., Unity or Unreal Engine). After saving is complete, the server sends a notification to the user. This notification includes the URL and login information for accessing the virtual environment platform.

[0483] The user logs into the virtual environment platform using the provided information. The user can access the virtual space using a VR headset or a PC and experience the recreated memories. Within this virtual environment, the user can walk around and interact with the recreated scenery and people.

[0484] Specific examples

[0485] Example of recreating family trip memories

[0486] 1. User uploads image data:

[0487] Users select photos taken during their summer vacation with their family on their smartphone and upload them using a dedicated application.

[0488] 2. Data storage by the server:

[0489] The server receives the uploaded photos and saves them in the specified storage path. After saving, the file path is passed to the generation AI.

[0490] 3. Analysis of image data and generation of 3D data:

[0491] The generative artificial intelligence analyzes and extracts images of family members and travel destinations from the photos, and generates three-dimensional virtual environment data based on this.

[0492] 4. 3D data storage and notification:

[0493] The server stores the three-dimensional data in the virtual environment platform, and after the storage is complete, notifies the user of the access information.

[0494] 5. Virtual Experience:

[0495] The user logs into the virtual environment platform using the provided information and uses a VR headset to experience the recreated memories.

[0496] Prompt Sentence Examples

[0497] "Recreate a beach memory using photos from your family summer vacation. Choose photos that show the whole family, including walking along the shore and playing in the ocean."

[0498] In this way, users can recreate their past memories in a virtual space and have a realistic experience, allowing them to vividly recall past experiences.

[0499] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0500] Step 1:

[0501] The user selects the image data and starts uploading.

[0502] Input: Image data (photos and videos) selected by the user using an application on a smartphone or PC.

[0503] How it works: The user launches the dedicated application, clicks the "Upload Photos" button, and selects the image data they want to upload from their device.

[0504] Output: The selected image data is saved to the device as an upload request.

[0505] Step 2:

[0506] The terminal transmits the image data to the server.

[0507] Input: Image data selected and saved by the user.

[0508] How it works: The device sends an upload request to the server, which uses its internet connection to split the image data into packets and begin sending them to the server.

[0509] Output: Packets of image data received by the server.

[0510] Step 3:

[0511] The server receives and stores the image data.

[0512] Input: Packets of image data sent from the device.

[0513] Operation: The server receives packets sequentially, integrates them, and saves them as complete image data in storage. After saving is complete, it records the path of the saved file.

[0514] Output: The saved image data and its file path in storage.

[0515] Step 4:

[0516] The server passes the file path to the generation AI.

[0517] Input: The file path of the saved image data.

[0518] Operation: The server generates a request to send the file path of the stored image data to the generating artificial intelligence and sends it over the network.

[0519] Output: The file path received by the generating AI.

[0520] Step 5:

[0521] Generative artificial intelligence analyzes image data and generates three-dimensional data.

[0522] Input: The file path of the received image data.

[0523] How it works: Generative AI (e.g., OpenAI's DALL-E or GPT model) analyzes image data based on file paths. The system extracts patterns of people, scenery, and movement within the image and generates 3D virtual environment data based on this data.

[0524] Output: Generated 3D virtual environment data.

[0525] Step 6:

[0526] The generated three-dimensional data is received by the server and stored in the virtual environment platform.

[0527] Input: 3D virtual environment data returned from a generative AI model.

[0528] How it works: The server receives the 3D data and uploads it to the virtual environment platform (e.g. Unity or Unreal Engine). It verifies the integrity of the data and checks that it has been saved correctly.

[0529] Output: Three-dimensional virtual environment data stored in the virtual environment platform.

[0530] Step 7:

[0531] The server sends a notification to the user.

[0532] Input: Three-dimensional virtual environment data stored in the virtual environment platform.

[0533] What it does: The server notifies the user that the save is complete. The notification includes a URL and login information for accessing the virtual environment platform.

[0534] Output: The notification the user receives, containing access information.

[0535] Step 8:

[0536] A user accesses the virtual environment platform.

[0537] Input: The URL and login information contained in the notification from the server.

[0538] Operation: The user accesses the virtual environment platform based on the notification and enters their login information. Once logged in, they can access the recreated space of their memories.

[0539] Output: Recreated memory data within the virtual environment accessed by the user.

[0540] Step 9:

[0541] Users can experience a memorable virtual space using VR.

[0542] Input: Recreated memory data in a virtual environment.

[0543] How it works: Using a VR headset or PC, users can walk around the virtual space and interact with the recreated scenery and people. Through the experience in the virtual space, they can relive past memories.

[0544] Output: An immersive virtual memory space experienced by the user.

[0545] Through the above processing steps, the user can re-experience the virtual space with a sense of realism based on the image data he or she has taken.

[0546] (Application example 1)

[0547] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0548] Simply storing a user's photos and videos visually is difficult to fully recreate the sense of presence and depth of memories. Furthermore, conventional systems make it difficult for users to move and interact within the virtual space, limiting the user experience. Thus, technology is needed to recreate real-world experiences in virtual spaces and provide a deeper sense of immersion.

[0549] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0550] In this invention, the server includes means for uploading media data selected by the user's terminal, means for receiving the uploaded media data and passing it to the generating AI, means for the generating AI to analyze the received media data and convert it into data for a three-dimensional virtual space, means for saving the converted three-dimensional data in the virtual space platform, means for the user to access the virtual space platform through the terminal and experience the recreated memories, means for the user to move and interact within the generated three-dimensional space, means for notifying the user of the three-dimensional data saved in the virtual space platform, and means for recreating the emotions and movements at the time selected by the user based on the analyzed media data, thereby enabling the user to have a more immersive virtual space experience.

[0551] "User terminal" refers to an electronic device used by a user, including a smartphone, tablet, or PC.

[0552] "Media Data" means data in digital form that contains visual or audio information, such as photographs or videos.

[0553] "Upload" is the process of sending data from a user's device to a server.

[0554] A "server" is a computer system installed to provide services over a network, and stores and manages data.

[0555] "Generative AI" refers to algorithms and models that analyze media data uploaded by users and generate new data.

[0556] "Analysis" is the process of examining received data in detail to identify and extract specific information.

[0557] "Data for three-dimensional virtual space" is digital data that has been converted into a format that can be visually displayed in three-dimensional space.

[0558] A "virtual space platform" is a software and hardware system that provides virtual reality (VR) and augmented reality (AR) environments.

[0559] "Notifying" is the process by which a system communicates some information to a user.

[0560] "Movement and interaction" refers to the user's actions of walking around the virtual space and interacting with other objects and characters.

[0561] "Reproducing emotions and movements" is the process of expressing a user's emotional state and physical movements at a particular point in time in a virtual space.

[0562] The present invention relates to a system for recreating memories in a virtual space based on media data such as photos and videos saved by a user. The system includes a user terminal, a server, a generating artificial intelligence, and a virtual space platform.

[0563] First, a dedicated application is installed on the user's device. The user uses this application to select and upload photos and videos from their device. An upload request is generated and the selected media data is sent to the server.

[0564] The server saves the uploaded media data in storage. Once saving is complete, it passes the file path to the generation AI. The generation AI receives this file path and begins analyzing the image data. This analysis uses computer vision technology (e.g., OpenCV) and machine learning models (e.g., TensorFlow). As a result of the analysis, the target person or object, background scenery, etc. are extracted.

[0565] The generative AI generates three-dimensional virtual space data based on the extracted information. This three-dimensional data is returned to the server and stored on the virtual space platform. After the server confirms that the data has been saved, it sends a notification to the user. This notification includes a URL and login information for accessing the virtual space platform.

[0566] The user logs into the virtual space platform using the provided access information. Using smart glasses or a head-mounted display, the user can access the generated virtual space. The user can walk around and interact within the virtual space. This allows the user to enjoy a realistic and immersive experience as a recreated memory.

[0567] For example, consider a specific scenario in which a virtual space is recreated based on photos taken on a family trip. The user opens a smartphone app, selects "family trip photos," and uploads them. The server receives the photos, performs image analysis, and generates a three-dimensional space. The user then logs in to the virtual space using the provided access information and puts on a VR headset, allowing them to re-experience the family trip in the created virtual space.

[0568] An example of a prompt to give to a generative AI model is:

[0569] "Create a virtual space that recreates family vacation memories from photos and videos uploaded by users. Identify specific landscapes, extract people's figures, and generate 3D models based on that data."

[0570] This system allows users to enjoy a more immersive virtual space experience as a recreated memory.

[0571] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0572] Step 1:

[0573] The user launches the smartphone application, selects saved photos and videos, generates an upload request for the selected media data, and sends the data to the server.

[0574] Input: Photos and videos stored on the user's device

[0575] Output: Upload request sent to the server

[0576] Specific operation: When a user selects a photo or video within the app and presses the "Upload" button, the media data is sent to the server.

[0577] Step 2:

[0578] The server receives an upload request from the user and saves the media data in storage. Once the saving is complete, the server passes the file path to the generation AI.

[0579] Input: User-uploaded photo and video data

[0580] Output: The file path saved in the storage

[0581] Specific operation: The server saves the media data in the file system, obtains the file path of the saved location, and provides it to the generating artificial intelligence.

[0582] Step 3:

[0583] The generative AI receives the file path and begins analyzing the image data. It uses libraries such as OpenCV and TensorFlow to extract the target person, object, background scenery, etc. As a result of the analysis, it generates data for a 3D virtual space.

[0584] Input: File path of media data stored in storage

[0585] Output: Data for 3D virtual space

[0586] Specific operation: The generative AI loads the image data, runs an analysis algorithm to extract the necessary information, and generates a three-dimensional model based on that information.

[0587] Step 4:

[0588] The server receives the 3D data generated by the AI ​​generator and stores it on the virtual space platform. After the data has been stored, the server sends a notification to the user. The notification includes access information to the virtual space platform (URL, login information, etc.).

[0589] Input: Data for 3D virtual space

[0590] Output: User notification

[0591] Specific operation: The server uploads the 3D data to the virtual space platform and sends a save completion notification to the user.

[0592] Step 5:

[0593] The user logs into the virtual space platform using the access information provided in the notification. The user accesses the virtual space by wearing a device such as smart glasses or a head-mounted display, and experiences the recreated memories. The user can walk around and interact with the virtual space.

[0594] Input: Access information (URL, login information), user interface device (smart glasses, head-mounted display, etc.)

[0595] Output: User experience in virtual space

[0596] Specific operations: The user logs into the virtual space platform using the provided URL and uses the VR device to explore the virtual space and interact with the recreated pictures and people.

[0597] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0598] The present invention is a system for recreating memories in a virtual space based on photos and videos saved by the user, and recognizing the user's emotions to enrich the recreating process. This system includes a user terminal, a server, a generative artificial intelligence, an emotion engine, and a virtual space platform.

[0599] First, the user's device has the function to upload selected image data (photos and videos). The user selects and uploads image data from their device through a dedicated application. An upload request is generated and the image data is sent to the server.

[0600] The server saves the received image data in storage. Once saving is complete, it passes the file path to the generating AI. The generating AI begins analyzing the image data and extracts the target scenery, people, movement patterns, etc. Based on this, data for the three-dimensional virtual space is generated.

[0601] Next, the emotion engine analyzes the image data and recognizes the user's emotions. Based on this recognized emotion information, the generative AI makes further fine adjustments. Specifically, it adjusts the environmental settings (e.g., lighting and music) and effects (visual effects such as warmth and coolness) in the virtual space based on the emotion information.

[0602] The generated 3D data and the data reflecting the emotion information are returned to the server and saved on the virtual space platform. After the server confirms that saving is complete, it sends a notification to the user. This notification includes the URL and login information for accessing the virtual space platform.

[0603] Users log in to the virtual space platform and access the recreated space using the URL they receive. Using a VR headset or PC, users can walk around the virtual space and interact with the scenery and people from their memories. Furthermore, the recreated space has environmental settings and effects that are adjusted based on the user's emotions, making for a more realistic and emotional experience.

[0604] Specific examples

[0605] Example of recreating family trip memories

[0606] We will explain an example of recreating a virtual space based on photos taken by a user on a summer vacation with their family, and also reflecting the emotions they felt at the time.

[0607] 1. Upload:

[0608] A user launches the application on their smartphone, clicks the "Upload Photos" button in the application, selects a few photos from a family trip, and uploads them to the server using a dedicated function in the application.

[0609] 2. Data storage:

[0610] The server receives the uploaded photo and saves it in storage. Once the saving is complete, the file path is passed to the AI ​​generator.

[0611] 3. Image and Sentiment Analysis:

[0612] The artificial intelligence system receives the file path and begins analyzing the image. The system extracts images of family members and scenery from travel destinations (such as the ocean or mountains), and generates data for a three-dimensional virtual space based on this information.

[0613] At the same time, the emotion engine analyzes the image data and recognizes the user's emotions (e.g., joy, surprise, excitement).

[0614] 4. Environment and Effects Settings:

[0615] Based on the emotional information recognized by the emotion engine, the generative AI adjusts the environmental settings (e.g., music and lighting) and effects (e.g., specific visual effects) in the virtual space, thereby reinforcing the emotions felt by the user at the time.

[0616] 5. Data Retention and Notification:

[0617] The server receives the generated 3D data and the data reflecting the emotional information, uploads it to the virtual space platform, and saves it. After saving is complete, the server notifies the user of the completion of saving and provides access information.

[0618] 6. Virtual Experience:

[0619] Users log in to the virtual space platform using the provided access information. They then put on a VR headset and can relive their family trip memories in the virtual space created. They can walk around, interact with the recreated scenery and people, and enjoy the environmental settings and effects that recreate the emotions of the time.

[0620] In this way, by recognizing the user's emotions and recreating memories based on those emotions, it is possible to provide a more realistic and moving experience.

[0621] The processing flow will be explained below.

[0622] Step 1:

[0623] The user launches the app and selects photos and videos stored on their device. When the user clicks the "Upload" button, the selected image data is sent to the server as an upload request.

[0624] Step 2:

[0625] The device generates an upload request, attaches the selected image data, and sends it to the server. The data is transferred using a secure communication protocol (e.g., HTTPS).

[0626] Step 3:

[0627] The server receives the upload request and saves the image data in storage. The saved data is registered in the database along with the file path.

[0628] Step 4:

[0629] The server passes the saved file path to the generation artificial intelligence (AI) module, which then reads the image data based on this file path.

[0630] Step 5:

[0631] Generative AI analyzes image data and detects scenery, people, and movements in photos and videos. Based on this, data for three-dimensional virtual space (3D models and animations) is generated.

[0632] Step 6:

[0633] The server passes the image data to the emotion engine, which then recognizes the user's emotion (e.g., joy, excitement, surprise) from the image data.

[0634] Step 7:

[0635] The emotion engine provides the recognized emotion information to the generation AI, which then adjusts the environmental settings (e.g., lighting, music) and visual effects in the virtual space based on this emotion information.

[0636] Step 8:

[0637] The artificial intelligence generates and adjusts the 3D data and environment setting data, which are then returned to the server, which then uploads and stores them on the virtual space platform.

[0638] Step 9:

[0639] The virtual space platform stores the uploaded data and generates a URL and login information that the user can access. The server receives the access information returned by the platform.

[0640] Step 10:

[0641] The server sends the user a URL and login information for accessing the virtual space platform via email or in-app notification.

[0642] Step 11:

[0643] The user receives the notification and accesses the specified URL, where they log in to the virtual space platform and enter the recreated space of their memories.

[0644] Step 12:

[0645] Users can walk around the virtual space using a VR headset or PC, interacting with the recreated scenery and people. Environmental settings and visual effects based on the user's emotional information further enrich the experience.

[0646] Through the above processing steps, a realistic and moving memory can be reproduced in a three-dimensional virtual space using the user's image data.

[0647] Example 2

[0648] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0649] Conventional technologies have limited means for users to recreate their individual memories in a virtual space based on photos and videos they have taken. It has also been difficult to provide a more realistic and moving experience by reflecting the user's emotions. Therefore, it is necessary to recognize the user's emotions and provide appropriate environmental settings and effects to enrich the recreation of memories in a virtual space.

[0650] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0651] In this invention, the server includes means for receiving uploaded image data and saving it in storage, means for passing the path of the saved image file to the generating AI, and means for saving the generated three-dimensional data and data reflecting emotional information in the virtual space platform. This makes it possible to generate and save a three-dimensional virtual space that takes emotional information into consideration based on image data taken by the user, and to provide the user with the experience of recreating moving memories.

[0652] "User terminal" refers to any electronic device that a user operates to send and receive data.

[0653] A "server" is a computer system that receives, stores, processes, and provides data over a network.

[0654] "Image data" refers to digital files that contain visual information, such as photographs and videos.

[0655] "Storage" refers to a physical or virtual medium for storing digital data.

[0656] "Generative AI" refers to artificial intelligence techniques for analyzing and generating data.

[0657] "Data for three-dimensional virtual space" refers to all digital information used to represent virtual space using three-dimensional computer graphics.

[0658] An "emotion engine" refers to technology that includes an algorithm for analyzing a user's emotions and generating an appropriate response based on that.

[0659] "Virtual space platform" refers to an integrated system of software and hardware that allows users to access and experience virtual spaces.

[0660] "Notification" refers to a message or alert that the system uses to communicate information to the user.

[0661] MODE FOR CARRYING OUT THE INVENTION

[0662] This invention is a system that recreates memories in a virtual space based on photos and videos saved by the user, and further enriches the recreation by recognizing the user's emotions. This system consists of a user terminal, a server, a generative artificial intelligence, an emotion engine, and a virtual space platform. The detailed processing of each hardware and software is described below.

[0663] Hardware and Software

[0664] User device: Refers to electronic devices such as smartphones, PCs, tablets, etc. Users select and upload image data using a dedicated application.

[0665] Server: A computer system that receives, stores, processes, and provides data. The server is equipped with storage and stores image data and generated 3D data.

[0666] Generative AI: AI technology that analyzes image data and generates 3D data. It extracts specific landscapes, people, and movement patterns to generate data for 3D virtual spaces.

[0667] Emotion engine: A technology that analyzes image data to recognize the user's emotions and generates environment settings and effects that reflect those emotions.

[0668] Virtual Space Platform: A system that allows users to access virtual spaces and relive memories. It can be accessed using a VR headset or a PC.

[0669] Program processing

[0670] Processing on the user's device

[0671] A user starts a dedicated application and selects and uploads photos and videos from their device. At this time, the device generates an upload request and sends the selected image data to the server. For example, a user selects and uploads several photos taken during a summer vacation with their family.

[0672] Processing on the server

[0673] The server receives the image data sent from the device and saves it in storage. Once the saving is complete, the server passes the file path to the AI ​​generator. For example, the server provides the AI ​​generator with the file path of a set of saved family trip photos.

[0674] Processing with generative artificial intelligence

[0675] The artificial intelligence begins analyzing the image data based on the file path received from the server. This allows it to extract the target scenery, people, and movement patterns, and generate data for a 3D virtual space. In this case, it recreates the beach scenery and family figures in 3D from photos of a family summer vacation.

[0676] Processing with the Emotion Engine

[0677] The emotion engine analyzes the image data and recognizes the user's emotions (e.g., joy, surprise, excitement). Based on this, the generative AI adjusts the environmental settings and effects of the virtual space. For example, if the user's emotion is recognized as "joy," the brightness and music in the virtual space will be adjusted.

[0678] Processing on a virtual space platform

[0679] The generated 3D data and the data reflecting the emotional information are sent back to the server and saved on the virtual space platform. Once saved, the server sends a notification to the user. The user can access the platform using the URL and login information in the notification and experience the memories recreated in the virtual space using a VR headset or PC.

[0680] Specific examples

[0681] Example of recreating family trip memories

[0682] An example of recreating a virtual space based on photos taken by a user on a summer vacation with their family would include the following steps:

[0683] 1. Upload

[0684] The user launches a dedicated app on their smartphone, selects a few photos from a family trip, and clicks the upload button.

[0685] 2. Data storage

[0686] The server receives the image data and stores it in storage.

[0687] 3. Image and Sentiment Analysis

[0688] Generative AI analyzes photos and an emotion engine recognizes emotions.

[0689] 4. Environment and Effects Settings

[0690] Adjusting environmental settings and effects in a virtual space based on emotional information.

[0691] 5. Data Retention and Notification

[0692] The server stores the generated data in the virtual space platform and notifies the user.

[0693] 6. Virtual Experience

[0694] Users use a VR headset to experience a virtual space that recreates memories of a family trip.

[0695] Prompt Sentence Examples

[0696] "Project: Memories of a summer family trip. Upload family photos on the screen. Then, we analyze emotions and add appropriate effects. For example, we create an environment that evokes feelings of joy or surprise."

[0697] This system allows users to recreate and experience memories more realistically based on the emotions they felt.

[0698] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0699] Step 1:

[0700] The user selects and uploads image data from the device using a dedicated application. Specifically, the user launches the smartphone app and clicks the "Upload Photos" button. They then select multiple photos from a family trip and press the upload button. The device generates an upload request based on this and sends the image data to the server.

[0701] Input: Image data selected by the user (photos, videos)

[0702] Output: Upload request to server

[0703] Step 2:

[0704] The server receives the image data sent from the terminal and stores it in storage. Specifically, the server stores the received image data file in a specified directory and registers the file path in the management system.

[0705] Input: Image data sent from the device

[0706] Output: Image file saved in storage, file path obtained

[0707] Step 3:

[0708] The server passes the file path of the stored image data to the generation AI. Specifically, the server provides the path of the image file as an input parameter of the generation AI and requests analysis.

[0709] Input: File path of saved image data

[0710] Output: Analysis request to the generative AI

[0711] Step 4:

[0712] Generative AI analyzes image data and extracts scenery, people, movement patterns, etc. For example, it can recognize scenery such as the sea and mountains, as well as family members' faces and movements from photos of a family trip, and use this information to generate data for a three-dimensional virtual space.

[0713] Input: Image data file path

[0714] Output: Data for 3D virtual space

[0715] Step 5:

[0716] The emotion engine analyzes image data and recognizes the user's emotions. Specifically, the emotion engine analyzes facial expressions and situations obtained from the photos to detect emotions such as joy and surprise. This emotional information is provided to the generative AI.

[0717] Input: Image data

[0718] Output: Recognized emotion information

[0719] Step 6:

[0720] The generative AI adjusts the environmental settings and effects in the virtual space based on the emotional information obtained from the emotion engine. Specifically, it selects the brightness and color of the lighting, background music, etc. based on the emotional information. For example, if joy is recognized, bright lighting and upbeat music will be set.

[0721] Input: Recognized emotion information

[0722] Output: Adjusted virtual space settings and effects

[0723] Step 7:

[0724] The server stores the data generated by the generative AI and emotion engine on the virtual space platform. Specifically, the server uploads the generated three-dimensional data and emotion information to the virtual space platform and stores them as data.

[0725] Input: 3D data, emotional information

[0726] Output: Data stored in the virtual space platform

[0727] Step 8:

[0728] The server notifies the user that the save has been completed. Specifically, the server sends a notification to the user's device that includes an access URL and login information.

[0729] Input: Completion information saved in the virtual space platform

[0730] Output: Notification to user (access URL, login information)

[0731] Step 9:

[0732] Based on the information provided, users can log in to the virtual space platform and experience the recreated memories. Specifically, users access the virtual space using a VR headset or PC and interact with the recreated scenery and people. They experience environmental settings and effects based on emotional information.

[0733] Input: Access URL, login information

[0734] Output: Experience in virtual space (recreated memory interaction)

[0735] (Application example 2)

[0736] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0737] Conventional virtual space reproduction systems based on photographs and videos have the problem that they are unable to create environmental settings and effects that take into account the user's emotions, resulting in a lack of realism in the experience. The present invention aims to solve this problem and provide a more emotionally rich and realistic virtual space experience.

[0738] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading image data selected by the user's terminal, means for the server to receive the uploaded image data and pass it to the generation AI, means for the generation AI to analyze the received image data and convert it into data for a three-dimensional virtual space, means for the emotion engine to analyze the image data and recognize the user's emotional information, means for the generation AI to adjust the environmental settings and effects of the virtual space based on the emotional information, means for the server to save the converted three-dimensional data in the virtual space platform, and means for the user to access the virtual space platform via their terminal and experience recreated memories. This enables a rich and realistic virtual space experience that reflects the user's emotions.

[0739] "User terminal" refers to a device used by a user to select and upload image data, and includes smartphones, tablets, PCs, etc.

[0740] "Image data" refers to data of photos and videos saved by the user, and is the basis for recreating the virtual space.

[0741] A "server" is a computer system that receives and stores uploaded image data, and also exchanges data with the generating artificial intelligence and virtual space platform.

[0742] "Generative AI" is an AI technology that analyzes image data and generates data for three-dimensional virtual space.

[0743] The "emotion engine" is a technology that recognizes the user's emotional information from analyzed image data and adjusts the environmental settings and effects of the virtual space based on that information.

[0744] A "virtual space platform" is a system that stores generated three-dimensional data and provides a virtual space for users to access and experience.

[0745] "Three-dimensional data" is data in a three-dimensional virtual space generated based on image data, and is data that represents objects and environments within the virtual space.

[0746] "Environmental settings" are settings that adjust environmental elements such as lighting and music within the virtual space.

[0747] "Effects" are visual and tactile effects that enrich the user's experience within the virtual space.

[0748]

[0749] The present invention is a system that recreates memories in a virtual space based on photos and videos saved by the user, and further enriches the recreation by recognizing the user's emotions. This system includes a user terminal, a server, a generative artificial intelligence, an emotion engine, and a virtual space platform.

[0750] First, the user's device has the function to upload selected image data (photos and videos). The user selects and uploads image data from their device through a dedicated application. An upload request is generated and the image data is sent to the server.

[0751] The server saves the received image data in storage. Once saving is complete, it passes the file path to the generating AI. The generating AI begins analyzing the image data and extracts the target scenery, people, movement patterns, etc. Based on this, data for the three-dimensional virtual space is generated.

[0752] Next, the emotion engine analyzes the image data and recognizes the user's emotions. Based on this recognized emotion information, the generative AI makes further fine adjustments. Specifically, it adjusts the environmental settings (e.g., lighting and music) and effects (visual effects such as warmth and coolness) in the virtual space based on the emotion information.

[0753] The generated 3D data and the data reflecting the emotional information are returned to the server and saved on the virtual space platform. After the server confirms that saving is complete, it sends a notification to the user. This notification includes the URL and login information for accessing the virtual space platform.

[0754] Users log in to the virtual space platform and access the recreated space using the URL they receive. Using a VR headset or PC, users can walk around the virtual space and interact with the scenery and people from their memories. Furthermore, the recreated space has environmental settings and effects that are adjusted based on the user's emotions, making for a more realistic and emotional experience.

[0755] Specific examples

[0756] Example of recreating family trip memories

[0757] 1. Upload:

[0758] A user launches the application on their smartphone, clicks the "Upload Photos" button in the application, selects a few photos from a family trip, and uploads them to the server using a dedicated function in the application.

[0759] 2. Data storage:

[0760] The server receives the uploaded photo and saves it in storage. Once the saving is complete, the file path is passed to the AI ​​generator.

[0761] 3. Image and Sentiment Analysis:

[0762] The generation AI receives the file path and begins image analysis. The system extracts images of family members and travel destinations (such as the ocean or mountains) and generates data for a 3D virtual space based on this. At the same time, the emotion engine analyzes the image data and recognizes the user's emotions (e.g., joy, surprise, excitement).

[0763] 4. Environment and Effects Settings:

[0764] Based on the emotional information recognized by the emotion engine, the generative AI adjusts the environmental settings (e.g., music and lighting) and effects (e.g., specific visual effects) in the virtual space, thereby reinforcing the emotions felt by the user at the time.

[0765] 5. Data Retention and Notification:

[0766] The server receives the generated 3D data and the data reflecting the emotional information, uploads it to the virtual space platform, and saves it. After saving is complete, the server notifies the user of the completion of saving and provides access information.

[0767] 6. Virtual Experience:

[0768] Users log in to the virtual space platform using the provided access information. They then put on a VR headset and can relive their family trip memories in the virtual space created. They can walk around, interact with the recreated scenery and people, and enjoy the environmental settings and effects that recreate the emotions of the time.

[0769] Prompt Sentence Examples

[0770] "I uploaded three photos and one video of my family at a carnival. Use these to create a virtual space filled with fun and excitement. Add colorful lights and lively music."

[0771] This system enables a rich and realistic virtual space experience that reflects the user's emotions.

[0772] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0773] Step 1:

[0774] The user launches the application on their smartphone and clicks the dedicated "Photo Upload" button. The user selects several photos and videos from a family trip from within the application and generates an upload request. At this time, the selected image data is imported into the application.

[0775] Step 2:

[0776] The user's device sends the generated upload request to the server. The server receives this request and saves the image data to storage. Once saving is complete, the file path is recorded. The image data received as input is saved and a file path is generated as output.

[0777] Step 3:

[0778] The server passes the saved file path to the generation AI. The generation AI receives the file path and begins analyzing the image data. It extracts the target scenery, people, and movement patterns from the image data, and generates data for a three-dimensional virtual space based on that. It receives image data as input and generates three-dimensional data as output.

[0779] Step 4:

[0780] The emotion engine analyzes the image data received from the generation AI and recognizes the user's emotional information (e.g., enjoyment, surprise, excitement). Based on the recognized emotional information, the generation AI adjusts the environmental settings (e.g., lighting and music) and effects (visual effects such as warmth and coolness) of the virtual space. It receives the user's emotional information as input and generates adjusted three-dimensional data as output.

[0781] Step 5:

[0782] The generated 3D data and the data reflecting the emotion information are returned to the server. The server uploads and saves this data to the virtual space platform. After confirming that saving is complete, the server sends a notification to the user. This notification includes the URL and login information for accessing the virtual space platform. It receives 3D data as input and generates a user notification as output.

[0783] Step 6:

[0784] The user logs into the virtual space platform using the notified URL and login information. Using a VR headset or PC, the user relives the memories of their family trip in the created virtual space. The user can walk around the virtual space and interact with the recreated scenery and people. The recreated space has environmental settings and effects adjusted based on emotions, making for a more realistic and moving experience. The notified information is used as input, and access to and experience of the virtual space is obtained as output.

[0785] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0786] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0787] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0788] [Third embodiment]

[0789] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0790] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0791] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0792] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0793] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0794] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0795] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0796] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0797] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0798] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0799] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0800] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0801] The present invention provides a system for recreating memories in a virtual space based on photos and videos saved by a user. The system includes a user terminal, a server, a generating artificial intelligence, and a virtual space platform.

[0802] First, the user's device has the function to upload selected image data (photos and videos). The user selects and uploads image data from their device through a dedicated application. An upload request is generated and the image data is sent to the server.

[0803] The server saves the received image data in storage. Once saving is complete, it passes the file path to the generating AI. The generating AI begins analyzing the image data and extracts the target scenery, people, movement patterns, etc. Based on this, data for the three-dimensional virtual space is generated.

[0804] The generated 3D data is returned to the server and stored in the virtual space platform. After the server confirms that the data has been saved, it sends a notification to the user. This notification includes the URL and login information for accessing the virtual space platform.

[0805] Users log in to the virtual space platform and use the URL to access the recreated space. Using a VR headset or PC, users can walk around the virtual space and interact with the scenery and people from their memories.

[0806] Specific examples

[0807] Example of recreating family trip memories

[0808] An example will be described in which a virtual space is recreated based on photos taken by a user on a summer vacation trip with his or her family.

[0809] 1. Upload:

[0810] A user launches the application on their smartphone, clicks the "Upload Photos" button in the application, selects a few photos from a family trip, and uploads them to the server using a dedicated function in the application.

[0811] 2. Data storage:

[0812] The server receives the uploaded photo and saves it in storage. Once the saving is complete, the file path is passed to the AI ​​generator.

[0813] 3. Image Analysis:

[0814] The artificial intelligence system receives the file path and begins analyzing the image. The system extracts images of family members and scenery from travel destinations (such as the ocean or mountains), and generates data for a three-dimensional virtual space based on this information.

[0815] 4. Data Retention and Notification:

[0816] The server receives the generated 3D data and uploads it to the virtual space platform for storage. After storage is complete, the server notifies the user of the completion of storage and provides access information.

[0817] 5. Virtual Experience:

[0818] The user logs into the virtual space platform using the provided access information. Putting on a VR headset, the user can relive memories of their family trip in the virtual space created. They can walk around, interact with the recreated scenery and people, and fully enjoy the experience.

[0819] In this way, this system allows users to recreate memories from their photos and videos in a realistic and immersive way. It can quickly respond to sudden changes in the environment and screen transitions, allowing users to relive their past memories without stress.

[0820] The processing flow will be explained below.

[0821] Step 1:

[0822] The user launches the app and selects photos and videos stored on their device. When the user clicks the "Upload" button, the selected image data is sent to the server as an upload request.

[0823] Step 2:

[0824] The device generates an upload request, attaches the selected image data, and sends it to the server. The data is transferred using a secure communication protocol (e.g., HTTPS).

[0825] Step 3:

[0826] The server receives the upload request and saves the image data in storage. The saved data is registered in the database along with the file path.

[0827] Step 4:

[0828] The server passes the saved file path to the generation artificial intelligence (AI) module, which then reads the image data based on this file path.

[0829] Step 5:

[0830] Generative AI analyzes image data to detect scenery, people, and movements in photos and videos. The analysis results are generated as three-dimensional data (e.g., 3D models, animations).

[0831] Step 6:

[0832] The generated 3D data is sent back from the AI ​​module to the server, which then receives the data and uploads it to the virtual space platform.

[0833] Step 7:

[0834] The virtual space platform stores the uploaded 3D data and generates a URL and login information that the user can access. The server receives the access information returned from the platform.

[0835] Step 8:

[0836] The server sends the user a URL and login information for accessing the virtual space platform via email or in-app notification.

[0837] Step 9:

[0838] The user receives the notification and accesses the specified URL, where they log in to the virtual space platform and enter the recreated space of their memories.

[0839] Step 10:

[0840] Users can use a VR headset or PC to walk around the virtual space and interact with the recreated scenery and people, allowing them to experience memories of that time in a realistic way.

[0841] Through the above processing steps, the user's image data is reproduced in a three-dimensional virtual space, allowing the user to re-experience memories in a realistic and immersive way.

[0842] Example 1

[0843] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0844] Conventional technologies for recreating memories based on photos and videos taken by users lack a sense of immersion, making it difficult to realistically recreate past experiences. Conventional systems simply display photos and videos on a flat screen, making it difficult to provide an experience that makes users feel as if they are actually there. This leads to a problem of a decrease in the excitement and realism felt when users relive past memories.

[0845] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0846] In this invention, the server includes a means for uploading image data selected by the user's information processing device, a means for the central processing unit to receive the uploaded image data and pass it to the generating AI, and a means for the generating AI to analyze the received image data and convert it into data for a three-dimensional virtual environment, thereby enabling the user to re-experience past memories in a more realistic and immersive virtual space.

[0847] A "user's information processing device" generally refers to a device that has computing resources for a user to operate, such as a smartphone or personal computer.

[0848] "Central Processing Unit" refers to a server or cloud service that has the computing resources to receive, store, and manage the analysis results of data.

[0849] "Generative AI" refers to an AI model that has the ability to analyze image data and generate three-dimensional virtual environment data based on the analysis results.

[0850] "Virtual environment platform" refers to the software and hardware infrastructure for storing generated three-dimensional virtual environment data and making it accessible to users.

[0851] "Image data" refers to visual information such as photos and videos uploaded by users.

[0852] "3D data" refers to 3D models and spatial information generated by generative artificial intelligence and usable within a virtual environment.

[0853] The term "uploading means" refers to a method or system for a user to operate an information processing device to transmit image data to a central processing unit.

[0854] The term "receiving means" refers to a method or system for the central processing unit to receive image data transmitted from the user's information processing device.

[0855] "Analysis means" refers to a method or system by which the generative artificial intelligence analyzes image data and converts it into three-dimensional data.

[0856] "Conversion means" refers to a method or system by which the generating artificial intelligence generates three-dimensional data based on image data.

[0857] "Storage means" refers to a method or system for recording the generated three-dimensional data in the virtual environment platform.

[0858] "Access means" refers to a method or system for a user to connect to the virtual environment platform via an information processing device and access a virtual environment generated based on the stored three-dimensional data.

[0859] This invention relates to a system that recreates memories in a virtual environment based on image data taken by the user. This system consists of a user's device, a server, a generative artificial intelligence (AI), and a virtual environment platform.

[0860] First, the user's information processing device (e.g., smartphone or personal computer) uploads image data (photos or videos). The user uses a dedicated application to select image data stored on the device and perform the upload operation. This operation causes the selected image data to be sent from the device to the server.

[0861] The server receives and saves the uploaded image data in its own storage. Once saved, the server passes the file path to a generative artificial intelligence (AI). The generative AI analyzes the image data using models such as OpenAI's DALL-E or GPT. This analysis extracts information such as people, scenery, and movement patterns from the image, and generates 3D virtual environment data.

[0862] The generated 3D data is returned to the server and saved in the virtual environment platform (e.g., Unity or Unreal Engine). After saving is complete, the server sends a notification to the user. This notification includes the URL and login information for accessing the virtual environment platform.

[0863] The user logs into the virtual environment platform using the provided information. The user can access the virtual space using a VR headset or a PC and experience the recreated memories. Within this virtual environment, the user can walk around and interact with the recreated scenery and people.

[0864] Specific examples

[0865] Example of recreating family trip memories

[0866] 1. User uploads image data:

[0867] Users select photos taken during their summer vacation with their family on their smartphone and upload them using a dedicated application.

[0868] 2. Data storage by the server:

[0869] The server receives the uploaded photos and saves them in the specified storage path. After saving, the file path is passed to the generation AI.

[0870] 3. Analysis of image data and generation of 3D data:

[0871] The generative artificial intelligence analyzes and extracts images of family members and travel destinations from the photos, and generates three-dimensional virtual environment data based on this.

[0872] 4. 3D data storage and notification:

[0873] The server stores the three-dimensional data in the virtual environment platform, and after the storage is complete, notifies the user of the access information.

[0874] 5. Virtual Experience:

[0875] The user logs into the virtual environment platform using the provided information and uses a VR headset to experience the recreated memories.

[0876] Prompt Sentence Examples

[0877] "Recreate a beach memory using photos from your family summer vacation. Choose photos that show the whole family, including walking along the shore and playing in the ocean."

[0878] In this way, users can recreate their past memories in a virtual space and have a realistic experience, allowing them to vividly recall past experiences.

[0879] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0880] Step 1:

[0881] The user selects the image data and starts uploading.

[0882] Input: Image data (photos and videos) selected by the user using an application on a smartphone or PC.

[0883] How it works: The user launches the dedicated application, clicks the "Upload Photos" button, and selects the image data they want to upload from their device.

[0884] Output: The selected image data is saved to the device as an upload request.

[0885] Step 2:

[0886] The terminal transmits the image data to the server.

[0887] Input: Image data selected and saved by the user.

[0888] How it works: The device sends an upload request to the server, which uses its internet connection to split the image data into packets and begin sending them to the server.

[0889] Output: Packets of image data received by the server.

[0890] Step 3:

[0891] The server receives and stores the image data.

[0892] Input: Packets of image data sent from the device.

[0893] Operation: The server receives packets sequentially, integrates them, and saves them as complete image data in storage. After saving is complete, it records the path of the saved file.

[0894] Output: The saved image data and its file path in storage.

[0895] Step 4:

[0896] The server passes the file path to the generation AI.

[0897] Input: The file path of the saved image data.

[0898] Operation: The server generates a request to send the file path of the stored image data to the generating artificial intelligence and sends it over the network.

[0899] Output: The file path received by the generating AI.

[0900] Step 5:

[0901] Generative artificial intelligence analyzes image data and generates three-dimensional data.

[0902] Input: The file path of the received image data.

[0903] How it works: Generative AI (e.g., OpenAI's DALL-E or GPT model) analyzes image data based on file paths. The system extracts patterns of people, scenery, and movement within the image and generates 3D virtual environment data based on this data.

[0904] Output: Generated 3D virtual environment data.

[0905] Step 6:

[0906] The generated three-dimensional data is received by the server and stored in the virtual environment platform.

[0907] Input: 3D virtual environment data returned from a generative AI model.

[0908] How it works: The server receives the 3D data and uploads it to the virtual environment platform (e.g. Unity or Unreal Engine). It verifies the integrity of the data and checks that it has been saved correctly.

[0909] Output: Three-dimensional virtual environment data stored in the virtual environment platform.

[0910] Step 7:

[0911] The server sends a notification to the user.

[0912] Input: Three-dimensional virtual environment data stored in the virtual environment platform.

[0913] What it does: The server notifies the user that the save is complete. The notification includes a URL and login information for accessing the virtual environment platform.

[0914] Output: The notification the user receives, containing access information.

[0915] Step 8:

[0916] A user accesses the virtual environment platform.

[0917] Input: The URL and login information contained in the notification from the server.

[0918] Operation: The user accesses the virtual environment platform based on the notification and enters their login information. Once logged in, they can access the recreated space of their memories.

[0919] Output: Recreated memory data within the virtual environment accessed by the user.

[0920] Step 9:

[0921] Users can experience a memorable virtual space using VR.

[0922] Input: Recreated memory data in a virtual environment.

[0923] How it works: Using a VR headset or PC, users can walk around the virtual space and interact with the recreated scenery and people. Through the experience in the virtual space, they can relive past memories.

[0924] Output: An immersive virtual memory space experienced by the user.

[0925] Through the above processing steps, the user can re-experience the virtual space with a sense of realism based on the image data he or she has taken.

[0926] (Application example 1)

[0927] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0928] Simply storing a user's photos and videos visually is difficult to fully recreate the sense of presence and depth of memories. Furthermore, conventional systems make it difficult for users to move and interact within the virtual space, limiting the user experience. Thus, technology is needed to recreate real-world experiences in virtual spaces and provide a deeper sense of immersion.

[0929] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0930] In this invention, the server includes means for uploading media data selected by the user's terminal, means for receiving the uploaded media data and passing it to the generating AI, means for the generating AI to analyze the received media data and convert it into data for a three-dimensional virtual space, means for saving the converted three-dimensional data in the virtual space platform, means for the user to access the virtual space platform through the terminal and experience the recreated memories, means for the user to move and interact within the generated three-dimensional space, means for notifying the user of the three-dimensional data saved in the virtual space platform, and means for recreating the emotions and movements at the time selected by the user based on the analyzed media data, thereby enabling the user to have a more immersive virtual space experience.

[0931] "User terminal" refers to an electronic device used by a user, including a smartphone, tablet, or PC.

[0932] "Media Data" means data in digital form that contains visual or audio information, such as photographs or videos.

[0933] "Upload" is the process of sending data from a user's device to a server.

[0934] A "server" is a computer system installed to provide services over a network, and stores and manages data.

[0935] "Generative AI" refers to algorithms and models that analyze media data uploaded by users and generate new data.

[0936] "Analysis" is the process of examining received data in detail to identify and extract specific information.

[0937] "Data for three-dimensional virtual space" is digital data that has been converted into a format that can be visually displayed in three-dimensional space.

[0938] A "virtual space platform" is a software and hardware system that provides virtual reality (VR) and augmented reality (AR) environments.

[0939] "Notifying" is the process by which a system communicates some information to a user.

[0940] "Movement and interaction" refers to the user's actions of walking around the virtual space and interacting with other objects and characters.

[0941] "Reproducing emotions and movements" is the process of expressing a user's emotional state and physical movements at a particular point in time in a virtual space.

[0942] The present invention relates to a system for recreating memories in a virtual space based on media data such as photos and videos saved by a user. The system includes a user terminal, a server, a generating artificial intelligence, and a virtual space platform.

[0943] First, a dedicated application is installed on the user's device. The user uses this application to select and upload photos and videos from their device. An upload request is generated and the selected media data is sent to the server.

[0944] The server saves the uploaded media data in storage. Once saving is complete, it passes the file path to the generation AI. The generation AI receives this file path and begins analyzing the image data. This analysis uses computer vision technology (e.g., OpenCV) and machine learning models (e.g., TensorFlow). As a result of the analysis, the target person or object, background scenery, etc. are extracted.

[0945] The generative AI generates three-dimensional virtual space data based on the extracted information. This three-dimensional data is returned to the server and stored on the virtual space platform. After the server confirms that the data has been saved, it sends a notification to the user. This notification includes a URL and login information for accessing the virtual space platform.

[0946] The user logs into the virtual space platform using the provided access information. Using smart glasses or a head-mounted display, the user can access the generated virtual space. The user can walk around and interact within the virtual space. This allows the user to enjoy a realistic and immersive experience as a recreated memory.

[0947] For example, consider a specific scenario in which a virtual space is recreated based on photos taken on a family trip. The user opens a smartphone app, selects "family trip photos," and uploads them. The server receives the photos, performs image analysis, and generates a three-dimensional space. The user then logs in to the virtual space using the provided access information and puts on a VR headset, allowing them to re-experience the family trip in the created virtual space.

[0948] An example of a prompt to give to a generative AI model is:

[0949] "Create a virtual space that recreates family vacation memories from photos and videos uploaded by users. Identify specific landscapes, extract people's figures, and generate 3D models based on that data."

[0950] This system allows users to enjoy a more immersive virtual space experience as a recreated memory.

[0951] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0952] Step 1:

[0953] The user launches the smartphone application, selects saved photos and videos, generates an upload request for the selected media data, and sends the data to the server.

[0954] Input: Photos and videos stored on the user's device

[0955] Output: Upload request sent to the server

[0956] Specific operation: When a user selects a photo or video within the app and presses the "Upload" button, the media data is sent to the server.

[0957] Step 2:

[0958] The server receives an upload request from the user and saves the media data in storage. Once the saving is complete, the server passes the file path to the generation AI.

[0959] Input: User-uploaded photo and video data

[0960] Output: The file path saved in the storage

[0961] Specific operation: The server saves the media data in the file system, obtains the file path of the saved location, and provides it to the generating artificial intelligence.

[0962] Step 3:

[0963] The generative AI receives the file path and begins analyzing the image data. It uses libraries such as OpenCV and TensorFlow to extract the target person, object, background scenery, etc. As a result of the analysis, it generates data for a 3D virtual space.

[0964] Input: File path of media data stored in storage

[0965] Output: Data for 3D virtual space

[0966] Specific operation: The generative AI loads the image data, runs an analysis algorithm to extract the necessary information, and generates a three-dimensional model based on that information.

[0967] Step 4:

[0968] The server receives the 3D data generated by the AI ​​generator and stores it on the virtual space platform. After the data has been stored, the server sends a notification to the user. The notification includes access information to the virtual space platform (URL, login information, etc.).

[0969] Input: Data for 3D virtual space

[0970] Output: User notification

[0971] Specific operation: The server uploads the 3D data to the virtual space platform and sends a save completion notification to the user.

[0972] Step 5:

[0973] The user logs into the virtual space platform using the access information provided in the notification. The user accesses the virtual space by wearing a device such as smart glasses or a head-mounted display, and experiences the recreated memories. The user can walk around and interact with the virtual space.

[0974] Input: Access information (URL, login information), user interface device (smart glasses, head-mounted display, etc.)

[0975] Output: User experience in virtual space

[0976] Specific operations: The user logs into the virtual space platform using the provided URL and uses the VR device to explore the virtual space and interact with the recreated pictures and people.

[0977] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0978] The present invention is a system for recreating memories in a virtual space based on photos and videos saved by the user, and recognizing the user's emotions to enrich the recreating process. This system includes a user terminal, a server, a generative artificial intelligence, an emotion engine, and a virtual space platform.

[0979] First, the user's device has the function to upload selected image data (photos and videos). The user selects and uploads image data from their device through a dedicated application. An upload request is generated and the image data is sent to the server.

[0980] The server saves the received image data in storage. Once saving is complete, it passes the file path to the generating AI. The generating AI begins analyzing the image data and extracts the target scenery, people, movement patterns, etc. Based on this, data for the three-dimensional virtual space is generated.

[0981] Next, the emotion engine analyzes the image data and recognizes the user's emotions. Based on this recognized emotion information, the generative AI makes further fine adjustments. Specifically, it adjusts the environmental settings (e.g., lighting and music) and effects (visual effects such as warmth and coolness) in the virtual space based on the emotion information.

[0982] The generated 3D data and the data reflecting the emotion information are returned to the server and saved on the virtual space platform. After the server confirms that saving is complete, it sends a notification to the user. This notification includes the URL and login information for accessing the virtual space platform.

[0983] Users log in to the virtual space platform and access the recreated space using the URL they receive. Using a VR headset or PC, users can walk around the virtual space and interact with the scenery and people from their memories. Furthermore, the recreated space has environmental settings and effects that are adjusted based on the user's emotions, making for a more realistic and emotional experience.

[0984] Specific examples

[0985] Example of recreating family trip memories

[0986] We will explain an example of recreating a virtual space based on photos taken by a user on a summer vacation with their family, and also reflecting the emotions they felt at the time.

[0987] 1. Upload:

[0988] A user launches the application on their smartphone, clicks the "Upload Photos" button in the application, selects a few photos from a family trip, and uploads them to the server using a dedicated function in the application.

[0989] 2. Data storage:

[0990] The server receives the uploaded photo and saves it in storage. Once the saving is complete, the file path is passed to the AI ​​generator.

[0991] 3. Image and Sentiment Analysis:

[0992] The artificial intelligence system receives the file path and begins analyzing the image. The system extracts images of family members and scenery from travel destinations (such as the ocean or mountains), and generates data for a three-dimensional virtual space based on this information.

[0993] At the same time, the emotion engine analyzes the image data and recognizes the user's emotions (e.g., joy, surprise, excitement).

[0994] 4. Environment and Effects Settings:

[0995] Based on the emotional information recognized by the emotion engine, the generative AI adjusts the environmental settings (e.g., music and lighting) and effects (e.g., specific visual effects) in the virtual space, thereby reinforcing the emotions felt by the user at the time.

[0996] 5. Data Retention and Notification:

[0997] The server receives the generated 3D data and the data reflecting the emotional information, uploads it to the virtual space platform, and saves it. After saving is complete, the server notifies the user of the completion of saving and provides access information.

[0998] 6. Virtual Experience:

[0999] Users log in to the virtual space platform using the provided access information. They then put on a VR headset and can relive their family trip memories in the virtual space created. They can walk around, interact with the recreated scenery and people, and enjoy the environmental settings and effects that recreate the emotions of the time.

[1000] In this way, by recognizing the user's emotions and recreating memories based on those emotions, it is possible to provide a more realistic and moving experience.

[1001] The processing flow will be explained below.

[1002] Step 1:

[1003] The user launches the app and selects photos and videos stored on their device. When the user clicks the "Upload" button, the selected image data is sent to the server as an upload request.

[1004] Step 2:

[1005] The device generates an upload request, attaches the selected image data, and sends it to the server. The data is transferred using a secure communication protocol (e.g., HTTPS).

[1006] Step 3:

[1007] The server receives the upload request and saves the image data in storage. The saved data is registered in the database along with the file path.

[1008] Step 4:

[1009] The server passes the saved file path to the generation artificial intelligence (AI) module, which then reads the image data based on this file path.

[1010] Step 5:

[1011] Generative AI analyzes image data and detects scenery, people, and movements in photos and videos. Based on this, data for three-dimensional virtual space (3D models and animations) is generated.

[1012] Step 6:

[1013] The server passes the image data to the emotion engine, which then recognizes the user's emotion (e.g., joy, excitement, surprise) from the image data.

[1014] Step 7:

[1015] The emotion engine provides the recognized emotion information to the generation AI, which then adjusts the environmental settings (e.g., lighting, music) and visual effects in the virtual space based on this emotion information.

[1016] Step 8:

[1017] The artificial intelligence generates and adjusts the 3D data and environment setting data, which are then returned to the server, which then uploads and stores them on the virtual space platform.

[1018] Step 9:

[1019] The virtual space platform stores the uploaded data and generates a URL and login information that the user can access. The server receives the access information returned by the platform.

[1020] Step 10:

[1021] The server sends the user a URL and login information for accessing the virtual space platform via email or in-app notification.

[1022] Step 11:

[1023] The user receives the notification and accesses the specified URL, where they log in to the virtual space platform and enter the recreated space of their memories.

[1024] Step 12:

[1025] Users can walk around the virtual space using a VR headset or PC, interacting with the recreated scenery and people. Environmental settings and visual effects based on the user's emotional information further enrich the experience.

[1026] Through the above processing steps, a realistic and moving memory can be reproduced in a three-dimensional virtual space using the user's image data.

[1027] Example 2

[1028] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1029] Conventional technologies have limited means for users to recreate their individual memories in a virtual space based on photos and videos they have taken. It has also been difficult to provide a more realistic and moving experience by reflecting the user's emotions. Therefore, it is necessary to recognize the user's emotions and provide appropriate environmental settings and effects to enrich the recreation of memories in a virtual space.

[1030] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1031] In this invention, the server includes means for receiving uploaded image data and saving it in storage, means for passing the path of the saved image file to the generating AI, and means for saving the generated three-dimensional data and data reflecting emotional information in the virtual space platform. This makes it possible to generate and save a three-dimensional virtual space that takes emotional information into consideration based on image data taken by the user, and to provide the user with the experience of recreating moving memories.

[1032] "User terminal" refers to any electronic device that a user operates to send and receive data.

[1033] A "server" is a computer system that receives, stores, processes, and provides data over a network.

[1034] "Image data" refers to digital files that contain visual information, such as photographs and videos.

[1035] "Storage" refers to a physical or virtual medium for storing digital data.

[1036] "Generative AI" refers to artificial intelligence techniques for analyzing and generating data.

[1037] "Data for three-dimensional virtual space" refers to all digital information used to represent virtual space using three-dimensional computer graphics.

[1038] An "emotion engine" refers to technology that includes an algorithm for analyzing a user's emotions and generating an appropriate response based on that.

[1039] "Virtual space platform" refers to an integrated system of software and hardware that allows users to access and experience virtual spaces.

[1040] "Notification" refers to a message or alert that the system uses to communicate information to the user.

[1041] MODE FOR CARRYING OUT THE INVENTION

[1042] This invention is a system that recreates memories in a virtual space based on photos and videos saved by the user, and further enriches the recreation by recognizing the user's emotions. This system consists of a user terminal, a server, a generative artificial intelligence, an emotion engine, and a virtual space platform. The detailed processing of each hardware and software is described below.

[1043] Hardware and Software

[1044] User device: Refers to electronic devices such as smartphones, PCs, tablets, etc. Users select and upload image data using a dedicated application.

[1045] Server: A computer system that receives, stores, processes, and provides data. The server is equipped with storage and stores image data and generated 3D data.

[1046] Generative AI: AI technology that analyzes image data and generates 3D data. It extracts specific landscapes, people, and movement patterns to generate data for 3D virtual spaces.

[1047] Emotion engine: A technology that analyzes image data to recognize the user's emotions and generates environment settings and effects that reflect those emotions.

[1048] Virtual Space Platform: A system that allows users to access virtual spaces and relive memories. It can be accessed using a VR headset or a PC.

[1049] Program processing

[1050] Processing on the user's device

[1051] A user starts a dedicated application and selects and uploads photos and videos from their device. At this time, the device generates an upload request and sends the selected image data to the server. For example, a user selects and uploads several photos taken during a summer vacation with their family.

[1052] Processing on the server

[1053] The server receives the image data sent from the device and saves it in storage. Once the saving is complete, the server passes the file path to the AI ​​generator. For example, the server provides the AI ​​generator with the file path of a set of saved family trip photos.

[1054] Processing with generative artificial intelligence

[1055] The artificial intelligence begins analyzing the image data based on the file path received from the server. This allows it to extract the target scenery, people, and movement patterns, and generate data for a 3D virtual space. In this case, it recreates the beach scenery and family figures in 3D from photos of a family summer vacation.

[1056] Processing with the Emotion Engine

[1057] The emotion engine analyzes the image data and recognizes the user's emotions (e.g., joy, surprise, excitement). Based on this, the generative AI adjusts the environmental settings and effects of the virtual space. For example, if the user's emotion is recognized as "joy," the brightness and music in the virtual space will be adjusted.

[1058] Processing on a virtual space platform

[1059] The generated 3D data and the data reflecting the emotional information are sent back to the server and saved on the virtual space platform. Once saved, the server sends a notification to the user. The user can access the platform using the URL and login information in the notification and experience the memories recreated in the virtual space using a VR headset or PC.

[1060] Specific examples

[1061] Example of recreating family trip memories

[1062] An example of recreating a virtual space based on photos taken by a user on a summer vacation with their family would include the following steps:

[1063] 1. Upload

[1064] The user launches a dedicated app on their smartphone, selects a few photos from a family trip, and clicks the upload button.

[1065] 2. Data storage

[1066] The server receives the image data and stores it in storage.

[1067] 3. Image and Sentiment Analysis

[1068] Generative AI analyzes photos and an emotion engine recognizes emotions.

[1069] 4. Environment and Effects Settings

[1070] Adjusting environmental settings and effects in a virtual space based on emotional information.

[1071] 5. Data Retention and Notification

[1072] The server stores the generated data in the virtual space platform and notifies the user.

[1073] 6. Virtual Experience

[1074] Users use a VR headset to experience a virtual space that recreates memories of a family trip.

[1075] Prompt Sentence Examples

[1076] "Project: Memories of a summer family trip. Upload family photos on the screen. Then, we analyze emotions and add appropriate effects. For example, we create an environment that evokes feelings of joy or surprise."

[1077] This system allows users to recreate and experience memories more realistically based on the emotions they felt.

[1078] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1079] Step 1:

[1080] The user selects and uploads image data from the device using a dedicated application. Specifically, the user launches the smartphone app and clicks the "Upload Photos" button. They then select multiple photos from a family trip and press the upload button. The device generates an upload request based on this and sends the image data to the server.

[1081] Input: Image data selected by the user (photos, videos)

[1082] Output: Upload request to server

[1083] Step 2:

[1084] The server receives the image data sent from the terminal and stores it in storage. Specifically, the server stores the received image data file in a specified directory and registers the file path in the management system.

[1085] Input: Image data sent from the device

[1086] Output: Image file saved in storage, file path obtained

[1087] Step 3:

[1088] The server passes the file path of the stored image data to the generation AI. Specifically, the server provides the path of the image file as an input parameter of the generation AI and requests analysis.

[1089] Input: File path of saved image data

[1090] Output: Analysis request to the generative AI

[1091] Step 4:

[1092] Generative AI analyzes image data and extracts scenery, people, movement patterns, etc. For example, it can recognize scenery such as the sea and mountains, as well as family members' faces and movements from photos of a family trip, and use this information to generate data for a three-dimensional virtual space.

[1093] Input: Image data file path

[1094] Output: Data for 3D virtual space

[1095] Step 5:

[1096] The emotion engine analyzes image data and recognizes the user's emotions. Specifically, the emotion engine analyzes facial expressions and situations obtained from the photos to detect emotions such as joy and surprise. This emotional information is provided to the generative AI.

[1097] Input: Image data

[1098] Output: Recognized emotion information

[1099] Step 6:

[1100] The generative AI adjusts the environmental settings and effects in the virtual space based on the emotional information obtained from the emotion engine. Specifically, it selects the brightness and color of the lighting, background music, etc. based on the emotional information. For example, if joy is recognized, bright lighting and upbeat music will be set.

[1101] Input: Recognized emotion information

[1102] Output: Adjusted virtual space settings and effects

[1103] Step 7:

[1104] The server stores the data generated by the generative AI and emotion engine on the virtual space platform. Specifically, the server uploads the generated three-dimensional data and emotion information to the virtual space platform and stores them as data.

[1105] Input: 3D data, emotional information

[1106] Output: Data stored in the virtual space platform

[1107] Step 8:

[1108] The server notifies the user that the save has been completed. Specifically, the server sends a notification to the user's device that includes an access URL and login information.

[1109] Input: Completion information saved in the virtual space platform

[1110] Output: Notification to user (access URL, login information)

[1111] Step 9:

[1112] Based on the information provided, users can log in to the virtual space platform and experience the recreated memories. Specifically, users access the virtual space using a VR headset or PC and interact with the recreated scenery and people. They experience environmental settings and effects based on emotional information.

[1113] Input: Access URL, login information

[1114] Output: Experience in virtual space (recreated memory interaction)

[1115] (Application example 2)

[1116] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1117] Conventional virtual space reproduction systems based on photographs and videos have the problem that they are unable to create environmental settings and effects that take into account the user's emotions, resulting in a lack of realism in the experience. The present invention aims to solve this problem and provide a more emotionally rich and realistic virtual space experience.

[1118] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading image data selected by the user's terminal, means for the server to receive the uploaded image data and pass it to the generation AI, means for the generation AI to analyze the received image data and convert it into data for a three-dimensional virtual space, means for the emotion engine to analyze the image data and recognize the user's emotional information, means for the generation AI to adjust the environmental settings and effects of the virtual space based on the emotional information, means for the server to save the converted three-dimensional data in the virtual space platform, and means for the user to access the virtual space platform via their terminal and experience recreated memories. This enables a rich and realistic virtual space experience that reflects the user's emotions.

[1119] "User terminal" refers to a device used by a user to select and upload image data, and includes smartphones, tablets, PCs, etc.

[1120] "Image data" refers to data of photos and videos saved by the user, and is the basis for recreating the virtual space.

[1121] A "server" is a computer system that receives and stores uploaded image data, and also exchanges data with the generating artificial intelligence and virtual space platform.

[1122] "Generative AI" is an AI technology that analyzes image data and generates data for three-dimensional virtual space.

[1123] The "emotion engine" is a technology that recognizes the user's emotional information from analyzed image data and adjusts the environmental settings and effects of the virtual space based on that information.

[1124] A "virtual space platform" is a system that stores generated three-dimensional data and provides a virtual space for users to access and experience.

[1125] "Three-dimensional data" is data in a three-dimensional virtual space generated based on image data, and is data that represents objects and environments within the virtual space.

[1126] "Environmental settings" are settings that adjust environmental elements such as lighting and music within the virtual space.

[1127] "Effects" are visual and tactile effects that enrich the user's experience within the virtual space.

[1128]

[1129] The present invention is a system that recreates memories in a virtual space based on photos and videos saved by the user, and further enriches the recreation by recognizing the user's emotions. This system includes a user terminal, a server, a generative artificial intelligence, an emotion engine, and a virtual space platform.

[1130] First, the user's device has the function to upload selected image data (photos and videos). The user selects and uploads image data from their device through a dedicated application. An upload request is generated and the image data is sent to the server.

[1131] The server saves the received image data in storage. Once saving is complete, it passes the file path to the generating AI. The generating AI begins analyzing the image data and extracts the target scenery, people, movement patterns, etc. Based on this, data for the three-dimensional virtual space is generated.

[1132] Next, the emotion engine analyzes the image data and recognizes the user's emotions. Based on this recognized emotion information, the generative AI makes further fine adjustments. Specifically, it adjusts the environmental settings (e.g., lighting and music) and effects (visual effects such as warmth and coolness) in the virtual space based on the emotion information.

[1133] The generated 3D data and the data reflecting the emotional information are returned to the server and saved on the virtual space platform. After the server confirms that saving is complete, it sends a notification to the user. This notification includes the URL and login information for accessing the virtual space platform.

[1134] Users log in to the virtual space platform and access the recreated space using the URL they receive. Using a VR headset or PC, users can walk around the virtual space and interact with the scenery and people from their memories. Furthermore, the recreated space has environmental settings and effects that are adjusted based on the user's emotions, making for a more realistic and emotional experience.

[1135] Specific examples

[1136] Example of recreating family trip memories

[1137] 1. Upload:

[1138] A user launches the application on their smartphone, clicks the "Upload Photos" button in the application, selects a few photos from a family trip, and uploads them to the server using a dedicated function in the application.

[1139] 2. Data storage:

[1140] The server receives the uploaded photo and saves it in storage. Once the saving is complete, the file path is passed to the AI ​​generator.

[1141] 3. Image and Sentiment Analysis:

[1142] The generation AI receives the file path and begins image analysis. The system extracts images of family members and travel destinations (such as the ocean or mountains) and generates data for a 3D virtual space based on this. At the same time, the emotion engine analyzes the image data and recognizes the user's emotions (e.g., joy, surprise, excitement).

[1143] 4. Environment and Effects Settings:

[1144] Based on the emotional information recognized by the emotion engine, the generative AI adjusts the environmental settings (e.g., music and lighting) and effects (e.g., specific visual effects) in the virtual space, thereby reinforcing the emotions felt by the user at the time.

[1145] 5. Data Retention and Notification:

[1146] The server receives the generated 3D data and the data reflecting the emotional information, uploads it to the virtual space platform, and saves it. After saving is complete, the server notifies the user of the completion of saving and provides access information.

[1147] 6. Virtual Experience:

[1148] Users log in to the virtual space platform using the provided access information. They then put on a VR headset and can relive their family trip memories in the virtual space created. They can walk around, interact with the recreated scenery and people, and enjoy the environmental settings and effects that recreate the emotions of the time.

[1149] Prompt Sentence Examples

[1150] "I uploaded three photos and one video of my family at a carnival. Use these to create a virtual space filled with fun and excitement. Add colorful lights and lively music."

[1151] This system enables a rich and realistic virtual space experience that reflects the user's emotions.

[1152] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1153] Step 1:

[1154] The user launches the application on their smartphone and clicks the dedicated "Photo Upload" button. The user selects several photos and videos from a family trip from within the application and generates an upload request. At this time, the selected image data is imported into the application.

[1155] Step 2:

[1156] The user's device sends the generated upload request to the server. The server receives this request and saves the image data to storage. Once saving is complete, the file path is recorded. The image data received as input is saved and a file path is generated as output.

[1157] Step 3:

[1158] The server passes the saved file path to the generation AI. The generation AI receives the file path and begins analyzing the image data. It extracts the target scenery, people, and movement patterns from the image data, and generates data for a three-dimensional virtual space based on that. It receives image data as input and generates three-dimensional data as output.

[1159] Step 4:

[1160] The emotion engine analyzes the image data received from the generation AI and recognizes the user's emotional information (e.g., enjoyment, surprise, excitement). Based on the recognized emotional information, the generation AI adjusts the environmental settings (e.g., lighting and music) and effects (visual effects such as warmth and coolness) of the virtual space. It receives the user's emotional information as input and generates adjusted three-dimensional data as output.

[1161] Step 5:

[1162] The generated 3D data and the data reflecting the emotion information are returned to the server. The server uploads and saves this data to the virtual space platform. After confirming that saving is complete, the server sends a notification to the user. This notification includes the URL and login information for accessing the virtual space platform. It receives 3D data as input and generates a user notification as output.

[1163] Step 6:

[1164] The user logs into the virtual space platform using the notified URL and login information. Using a VR headset or PC, the user relives the memories of their family trip in the created virtual space. The user can walk around the virtual space and interact with the recreated scenery and people. The recreated space has environmental settings and effects adjusted based on emotions, making for a more realistic and moving experience. The notified information is used as input, and access to and experience of the virtual space is obtained as output.

[1165] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1166] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1167] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1168] [Fourth embodiment]

[1169] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1170] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1171] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1172] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1173] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1174] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1175] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1176] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1177] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1178] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1179] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1180] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1181] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1182] The present invention provides a system for recreating memories in a virtual space based on photos and videos saved by a user. The system includes a user terminal, a server, a generating artificial intelligence, and a virtual space platform.

[1183] First, the user's device has the function to upload selected image data (photos and videos). The user selects and uploads image data from their device through a dedicated application. An upload request is generated and the image data is sent to the server.

[1184] The server saves the received image data in storage. Once saving is complete, it passes the file path to the generating AI. The generating AI begins analyzing the image data and extracts the target scenery, people, movement patterns, etc. Based on this, data for the three-dimensional virtual space is generated.

[1185] The generated 3D data is returned to the server and stored in the virtual space platform. After the server confirms that the data has been saved, it sends a notification to the user. This notification includes the URL and login information for accessing the virtual space platform.

[1186] Users log in to the virtual space platform and use the URL to access the recreated space. Using a VR headset or PC, users can walk around the virtual space and interact with the scenery and people from their memories.

[1187] Specific examples

[1188] Example of recreating family trip memories

[1189] An example will be described in which a virtual space is recreated based on photos taken by a user on a summer vacation trip with his or her family.

[1190] 1. Upload:

[1191] A user launches the application on their smartphone, clicks the "Upload Photos" button in the application, selects a few photos from a family trip, and uploads them to the server using a dedicated function in the application.

[1192] 2. Data storage:

[1193] The server receives the uploaded photo and saves it in storage. Once the saving is complete, the file path is passed to the AI ​​generator.

[1194] 3. Image Analysis:

[1195] The artificial intelligence system receives the file path and begins analyzing the image. The system extracts images of family members and scenery from travel destinations (such as the ocean or mountains), and generates data for a three-dimensional virtual space based on this information.

[1196] 4. Data Retention and Notification:

[1197] The server receives the generated 3D data and uploads it to the virtual space platform for storage. After storage is complete, the server notifies the user of the completion of storage and provides access information.

[1198] 5. Virtual Experience:

[1199] The user logs into the virtual space platform using the provided access information. Putting on a VR headset, the user can relive memories of their family trip in the virtual space created. They can walk around, interact with the recreated scenery and people, and fully enjoy the experience.

[1200] In this way, this system allows users to recreate memories from their photos and videos in a realistic and immersive way. It can quickly respond to sudden changes in the environment and screen transitions, allowing users to relive their past memories without stress.

[1201] The processing flow will be explained below.

[1202] Step 1:

[1203] The user launches the app and selects photos and videos stored on their device. When the user clicks the "Upload" button, the selected image data is sent to the server as an upload request.

[1204] Step 2:

[1205] The device generates an upload request, attaches the selected image data, and sends it to the server. The data is transferred using a secure communication protocol (e.g., HTTPS).

[1206] Step 3:

[1207] The server receives the upload request and saves the image data in storage. The saved data is registered in the database along with the file path.

[1208] Step 4:

[1209] The server passes the saved file path to the generation artificial intelligence (AI) module, which then reads the image data based on this file path.

[1210] Step 5:

[1211] Generative AI analyzes image data to detect scenery, people, and movements in photos and videos. The analysis results are generated as three-dimensional data (e.g., 3D models, animations).

[1212] Step 6:

[1213] The generated 3D data is sent back from the AI ​​module to the server, which then receives the data and uploads it to the virtual space platform.

[1214] Step 7:

[1215] The virtual space platform stores the uploaded 3D data and generates a URL and login information that the user can access. The server receives the access information returned from the platform.

[1216] Step 8:

[1217] The server sends the user a URL and login information for accessing the virtual space platform via email or in-app notification.

[1218] Step 9:

[1219] The user receives the notification and accesses the specified URL, where they log in to the virtual space platform and enter the recreated space of their memories.

[1220] Step 10:

[1221] Users can use a VR headset or PC to walk around the virtual space and interact with the recreated scenery and people, allowing them to experience memories of that time in a realistic way.

[1222] Through the above processing steps, the user's image data is reproduced in a three-dimensional virtual space, allowing the user to re-experience memories in a realistic and immersive way.

[1223] Example 1

[1224] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1225] Conventional technologies for recreating memories based on photos and videos taken by users lack a sense of immersion, making it difficult to realistically recreate past experiences. Conventional systems simply display photos and videos on a flat screen, making it difficult to provide an experience that makes users feel as if they are actually there. This leads to a problem of a decrease in the excitement and realism felt when users relive past memories.

[1226] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1227] In this invention, the server includes a means for uploading image data selected by the user's information processing device, a means for the central processing unit to receive the uploaded image data and pass it to the generating AI, and a means for the generating AI to analyze the received image data and convert it into data for a three-dimensional virtual environment, thereby enabling the user to re-experience past memories in a more realistic and immersive virtual space.

[1228] A "user's information processing device" generally refers to a device that has computing resources for a user to operate, such as a smartphone or personal computer.

[1229] "Central Processing Unit" refers to a server or cloud service that has the computing resources to receive, store, and manage the analysis results of data.

[1230] "Generative AI" refers to an AI model that has the ability to analyze image data and generate three-dimensional virtual environment data based on the analysis results.

[1231] "Virtual environment platform" refers to the software and hardware infrastructure for storing generated three-dimensional virtual environment data and making it accessible to users.

[1232] "Image data" refers to visual information such as photos and videos uploaded by users.

[1233] "3D data" refers to 3D models and spatial information generated by generative artificial intelligence and usable within a virtual environment.

[1234] The term "uploading means" refers to a method or system for a user to operate an information processing device to transmit image data to a central processing unit.

[1235] The term "receiving means" refers to a method or system for the central processing unit to receive image data transmitted from the user's information processing device.

[1236] "Analysis means" refers to a method or system by which the generative artificial intelligence analyzes image data and converts it into three-dimensional data.

[1237] "Conversion means" refers to a method or system by which the generating artificial intelligence generates three-dimensional data based on image data.

[1238] "Storage means" refers to a method or system for recording the generated three-dimensional data in the virtual environment platform.

[1239] "Access means" refers to a method or system for a user to connect to the virtual environment platform via an information processing device and access a virtual environment generated based on the stored three-dimensional data.

[1240] This invention relates to a system that recreates memories in a virtual environment based on image data taken by the user. This system consists of a user's device, a server, a generative artificial intelligence (AI), and a virtual environment platform.

[1241] First, the user's information processing device (e.g., smartphone or personal computer) uploads image data (photos or videos). The user uses a dedicated application to select image data stored on the device and perform the upload operation. This operation causes the selected image data to be sent from the device to the server.

[1242] The server receives and saves the uploaded image data in its own storage. Once saved, the server passes the file path to a generative artificial intelligence (AI). The generative AI analyzes the image data using models such as OpenAI's DALL-E or GPT. This analysis extracts information such as people, scenery, and movement patterns from the image, and generates 3D virtual environment data.

[1243] The generated 3D data is returned to the server and saved in the virtual environment platform (e.g., Unity or Unreal Engine). After saving is complete, the server sends a notification to the user. This notification includes the URL and login information for accessing the virtual environment platform.

[1244] The user logs into the virtual environment platform using the provided information. The user can access the virtual space using a VR headset or a PC and experience the recreated memories. Within this virtual environment, the user can walk around and interact with the recreated scenery and people.

[1245] Specific examples

[1246] Example of recreating family trip memories

[1247] 1. User uploads image data:

[1248] Users select photos taken during their summer vacation with their family on their smartphone and upload them using a dedicated application.

[1249] 2. Data storage by the server:

[1250] The server receives the uploaded photos and saves them in the specified storage path. After saving, the file path is passed to the generation AI.

[1251] 3. Analysis of image data and generation of 3D data:

[1252] The generative artificial intelligence analyzes and extracts images of family members and travel destinations from the photos, and generates three-dimensional virtual environment data based on this.

[1253] 4. 3D data storage and notification:

[1254] The server stores the three-dimensional data in the virtual environment platform, and after the storage is complete, notifies the user of the access information.

[1255] 5. Virtual Experience:

[1256] The user logs into the virtual environment platform using the provided information and uses a VR headset to experience the recreated memories.

[1257] Prompt Sentence Examples

[1258] "Recreate a beach memory using photos from your family summer vacation. Choose photos that show the whole family, including walking along the shore and playing in the ocean."

[1259] In this way, users can recreate their past memories in a virtual space and have a realistic experience, allowing them to vividly recall past experiences.

[1260] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1261] Step 1:

[1262] The user selects the image data and starts uploading.

[1263] Input: Image data (photos and videos) selected by the user using an application on a smartphone or PC.

[1264] How it works: The user launches the dedicated application, clicks the "Upload Photos" button, and selects the image data they want to upload from their device.

[1265] Output: The selected image data is saved to the device as an upload request.

[1266] Step 2:

[1267] The terminal transmits the image data to the server.

[1268] Input: Image data selected and saved by the user.

[1269] How it works: The device sends an upload request to the server, which uses its internet connection to split the image data into packets and begin sending them to the server.

[1270] Output: Packets of image data received by the server.

[1271] Step 3:

[1272] The server receives and stores the image data.

[1273] Input: Packets of image data sent from the device.

[1274] Operation: The server receives packets sequentially, integrates them, and saves them as complete image data in storage. After saving is complete, it records the path of the saved file.

[1275] Output: The saved image data and its file path in storage.

[1276] Step 4:

[1277] The server passes the file path to the generation AI.

[1278] Input: The file path of the saved image data.

[1279] Operation: The server generates a request to send the file path of the stored image data to the generating artificial intelligence and sends it over the network.

[1280] Output: The file path received by the generating AI.

[1281] Step 5:

[1282] Generative artificial intelligence analyzes image data and generates three-dimensional data.

[1283] Input: The file path of the received image data.

[1284] How it works: Generative AI (e.g., OpenAI's DALL-E or GPT model) analyzes image data based on file paths. The system extracts patterns of people, scenery, and movement within the image and generates 3D virtual environment data based on this data.

[1285] Output: Generated 3D virtual environment data.

[1286] Step 6:

[1287] The generated three-dimensional data is received by the server and stored in the virtual environment platform.

[1288] Input: 3D virtual environment data returned from a generative AI model.

[1289] How it works: The server receives the 3D data and uploads it to the virtual environment platform (e.g. Unity or Unreal Engine). It verifies the integrity of the data and checks that it has been saved correctly.

[1290] Output: Three-dimensional virtual environment data stored in the virtual environment platform.

[1291] Step 7:

[1292] The server sends a notification to the user.

[1293] Input: Three-dimensional virtual environment data stored in the virtual environment platform.

[1294] What it does: The server notifies the user that the save is complete. The notification includes a URL and login information for accessing the virtual environment platform.

[1295] Output: The notification the user receives, containing access information.

[1296] Step 8:

[1297] A user accesses the virtual environment platform.

[1298] Input: The URL and login information contained in the notification from the server.

[1299] Operation: The user accesses the virtual environment platform based on the notification and enters their login information. Once logged in, they can access the recreated space of their memories.

[1300] Output: Recreated memory data within the virtual environment accessed by the user.

[1301] Step 9:

[1302] Users can experience a memorable virtual space using VR.

[1303] Input: Recreated memory data in a virtual environment.

[1304] How it works: Using a VR headset or PC, users can walk around the virtual space and interact with the recreated scenery and people. Through the experience in the virtual space, they can relive past memories.

[1305] Output: An immersive virtual memory space experienced by the user.

[1306] Through the above processing steps, the user can re-experience the virtual space with a sense of realism based on the image data he or she has taken.

[1307] (Application example 1)

[1308] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1309] Simply storing a user's photos and videos visually is difficult to fully recreate the sense of presence and depth of memories. Furthermore, conventional systems make it difficult for users to move and interact within the virtual space, limiting the user experience. Thus, technology is needed to recreate real-world experiences in virtual spaces and provide a deeper sense of immersion.

[1310] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1311] In this invention, the server includes means for uploading media data selected by the user's terminal, means for receiving the uploaded media data and passing it to the generating AI, means for the generating AI to analyze the received media data and convert it into data for a three-dimensional virtual space, means for saving the converted three-dimensional data in the virtual space platform, means for the user to access the virtual space platform through the terminal and experience the recreated memories, means for the user to move and interact within the generated three-dimensional space, means for notifying the user of the three-dimensional data saved in the virtual space platform, and means for recreating the emotions and movements at the time selected by the user based on the analyzed media data, thereby enabling the user to have a more immersive virtual space experience.

[1312] "User terminal" refers to an electronic device used by a user, including a smartphone, tablet, or PC.

[1313] "Media Data" means data in digital form that contains visual or audio information, such as photographs or videos.

[1314] "Upload" is the process of sending data from a user's device to a server.

[1315] A "server" is a computer system installed to provide services over a network, and stores and manages data.

[1316] "Generative AI" refers to algorithms and models that analyze media data uploaded by users and generate new data.

[1317] "Analysis" is the process of examining received data in detail to identify and extract specific information.

[1318] "Data for three-dimensional virtual space" is digital data that has been converted into a format that can be visually displayed in three-dimensional space.

[1319] A "virtual space platform" is a software and hardware system that provides virtual reality (VR) and augmented reality (AR) environments.

[1320] "Notifying" is the process by which a system communicates some information to a user.

[1321] "Movement and interaction" refers to the user's actions of walking around the virtual space and interacting with other objects and characters.

[1322] "Reproducing emotions and movements" is the process of expressing a user's emotional state and physical movements at a particular point in time in a virtual space.

[1323] The present invention relates to a system for recreating memories in a virtual space based on media data such as photos and videos saved by a user. The system includes a user terminal, a server, a generating artificial intelligence, and a virtual space platform.

[1324] First, a dedicated application is installed on the user's device. The user uses this application to select and upload photos and videos from their device. An upload request is generated and the selected media data is sent to the server.

[1325] The server saves the uploaded media data in storage. Once saving is complete, it passes the file path to the generation AI. The generation AI receives this file path and begins analyzing the image data. This analysis uses computer vision technology (e.g., OpenCV) and machine learning models (e.g., TensorFlow). As a result of the analysis, the target person or object, background scenery, etc. are extracted.

[1326] The generative AI generates three-dimensional virtual space data based on the extracted information. This three-dimensional data is returned to the server and stored on the virtual space platform. After the server confirms that the data has been saved, it sends a notification to the user. This notification includes a URL and login information for accessing the virtual space platform.

[1327] The user logs into the virtual space platform using the provided access information. Using smart glasses or a head-mounted display, the user can access the generated virtual space. The user can walk around and interact within the virtual space. This allows the user to enjoy a realistic and immersive experience as a recreated memory.

[1328] For example, consider a specific scenario in which a virtual space is recreated based on photos taken on a family trip. The user opens a smartphone app, selects "family trip photos," and uploads them. The server receives the photos, performs image analysis, and generates a three-dimensional space. The user then logs in to the virtual space using the provided access information and puts on a VR headset, allowing them to re-experience the family trip in the created virtual space.

[1329] An example of a prompt to give to a generative AI model is:

[1330] "Create a virtual space that recreates family vacation memories from photos and videos uploaded by users. Identify specific landscapes, extract people's figures, and generate 3D models based on that data."

[1331] This system allows users to enjoy a more immersive virtual space experience as a recreated memory.

[1332] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1333] Step 1:

[1334] The user launches the smartphone application, selects saved photos and videos, generates an upload request for the selected media data, and sends the data to the server.

[1335] Input: Photos and videos stored on the user's device

[1336] Output: Upload request sent to the server

[1337] Specific operation: When a user selects a photo or video within the app and presses the "Upload" button, the media data is sent to the server.

[1338] Step 2:

[1339] The server receives an upload request from the user and saves the media data in storage. Once the saving is complete, the server passes the file path to the generation AI.

[1340] Input: User-uploaded photo and video data

[1341] Output: The file path saved in the storage

[1342] Specific operation: The server saves the media data in the file system, obtains the file path of the saved location, and provides it to the generating artificial intelligence.

[1343] Step 3:

[1344] The generative AI receives the file path and begins analyzing the image data. It uses libraries such as OpenCV and TensorFlow to extract the target person, object, background scenery, etc. As a result of the analysis, it generates data for a 3D virtual space.

[1345] Input: File path of media data stored in storage

[1346] Output: Data for 3D virtual space

[1347] Specific operation: The generative AI loads the image data, runs an analysis algorithm to extract the necessary information, and generates a three-dimensional model based on that information.

[1348] Step 4:

[1349] The server receives the 3D data generated by the AI ​​generator and stores it on the virtual space platform. After the data has been stored, the server sends a notification to the user. The notification includes access information to the virtual space platform (URL, login information, etc.).

[1350] Input: Data for 3D virtual space

[1351] Output: User notification

[1352] Specific operation: The server uploads the 3D data to the virtual space platform and sends a save completion notification to the user.

[1353] Step 5:

[1354] The user logs into the virtual space platform using the access information provided in the notification. The user accesses the virtual space by wearing a device such as smart glasses or a head-mounted display, and experiences the recreated memories. The user can walk around and interact with the virtual space.

[1355] Input: Access information (URL, login information), user interface device (smart glasses, head-mounted display, etc.)

[1356] Output: User experience in virtual space

[1357] Specific operations: The user logs into the virtual space platform using the provided URL and uses the VR device to explore the virtual space and interact with the recreated pictures and people.

[1358] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1359] The present invention is a system for recreating memories in a virtual space based on photos and videos saved by the user, and recognizing the user's emotions to enrich the recreating process. This system includes a user terminal, a server, a generative artificial intelligence, an emotion engine, and a virtual space platform.

[1360] First, the user's device has the function to upload selected image data (photos and videos). The user selects and uploads image data from their device through a dedicated application. An upload request is generated and the image data is sent to the server.

[1361] The server saves the received image data in storage. Once saving is complete, it passes the file path to the generating AI. The generating AI begins analyzing the image data and extracts the target scenery, people, movement patterns, etc. Based on this, data for the three-dimensional virtual space is generated.

[1362] Next, the emotion engine analyzes the image data and recognizes the user's emotions. Based on this recognized emotion information, the generative AI makes further fine adjustments. Specifically, it adjusts the environmental settings (e.g., lighting and music) and effects (visual effects such as warmth and coolness) in the virtual space based on the emotion information.

[1363] The generated 3D data and the data reflecting the emotion information are returned to the server and saved on the virtual space platform. After the server confirms that saving is complete, it sends a notification to the user. This notification includes the URL and login information for accessing the virtual space platform.

[1364] Users log in to the virtual space platform and access the recreated space using the URL they receive. Using a VR headset or PC, users can walk around the virtual space and interact with the scenery and people from their memories. Furthermore, the recreated space has environmental settings and effects that are adjusted based on the user's emotions, making for a more realistic and emotional experience.

[1365] Specific examples

[1366] Example of recreating family trip memories

[1367] We will explain an example of recreating a virtual space based on photos taken by a user on a summer vacation with their family, and also reflecting the emotions they felt at the time.

[1368] 1. Upload:

[1369] A user launches the application on their smartphone, clicks the "Upload Photos" button in the application, selects a few photos from a family trip, and uploads them to the server using a dedicated function in the application.

[1370] 2. Data storage:

[1371] The server receives the uploaded photo and saves it in storage. Once the saving is complete, the file path is passed to the AI ​​generator.

[1372] 3. Image and Sentiment Analysis:

[1373] The artificial intelligence system receives the file path and begins analyzing the image. The system extracts images of family members and scenery from travel destinations (such as the ocean or mountains), and generates data for a three-dimensional virtual space based on this information.

[1374] At the same time, the emotion engine analyzes the image data and recognizes the user's emotions (e.g., joy, surprise, excitement).

[1375] 4. Environment and Effects Settings:

[1376] Based on the emotional information recognized by the emotion engine, the generative AI adjusts the environmental settings (e.g., music and lighting) and effects (e.g., specific visual effects) in the virtual space, thereby reinforcing the emotions felt by the user at the time.

[1377] 5. Data Retention and Notification:

[1378] The server receives the generated 3D data and the data reflecting the emotional information, uploads it to the virtual space platform, and saves it. After saving is complete, the server notifies the user of the completion of saving and provides access information.

[1379] 6. Virtual Experience:

[1380] Users log in to the virtual space platform using the provided access information. They then put on a VR headset and can relive their family trip memories in the virtual space created. They can walk around, interact with the recreated scenery and people, and enjoy the environmental settings and effects that recreate the emotions of the time.

[1381] In this way, by recognizing the user's emotions and recreating memories based on those emotions, it is possible to provide a more realistic and moving experience.

[1382] The processing flow will be explained below.

[1383] Step 1:

[1384] The user launches the app and selects photos and videos stored on their device. When the user clicks the "Upload" button, the selected image data is sent to the server as an upload request.

[1385] Step 2:

[1386] The device generates an upload request, attaches the selected image data, and sends it to the server. The data is transferred using a secure communication protocol (e.g., HTTPS).

[1387] Step 3:

[1388] The server receives the upload request and saves the image data in storage. The saved data is registered in the database along with the file path.

[1389] Step 4:

[1390] The server passes the saved file path to the generation artificial intelligence (AI) module, which then reads the image data based on this file path.

[1391] Step 5:

[1392] Generative AI analyzes image data and detects scenery, people, and movements in photos and videos. Based on this, data for three-dimensional virtual space (3D models and animations) is generated.

[1393] Step 6:

[1394] The server passes the image data to the emotion engine, which then recognizes the user's emotion (e.g., joy, excitement, surprise) from the image data.

[1395] Step 7:

[1396] The emotion engine provides the recognized emotion information to the generation AI, which then adjusts the environmental settings (e.g., lighting, music) and visual effects in the virtual space based on this emotion information.

[1397] Step 8:

[1398] The artificial intelligence generates and adjusts the 3D data and environment setting data, which are then returned to the server, which then uploads and stores them on the virtual space platform.

[1399] Step 9:

[1400] The virtual space platform stores the uploaded data and generates a URL and login information that the user can access. The server receives the access information returned by the platform.

[1401] Step 10:

[1402] The server sends the user a URL and login information for accessing the virtual space platform via email or in-app notification.

[1403] Step 11:

[1404] The user receives the notification and accesses the specified URL, where they log in to the virtual space platform and enter the recreated space of their memories.

[1405] Step 12:

[1406] Users can walk around the virtual space using a VR headset or PC, interacting with the recreated scenery and people. Environmental settings and visual effects based on the user's emotional information further enrich the experience.

[1407] Through the above processing steps, a realistic and moving memory can be reproduced in a three-dimensional virtual space using the user's image data.

[1408] Example 2

[1409] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1410] Conventional technologies have limited means for users to recreate their individual memories in a virtual space based on photos and videos they have taken. It has also been difficult to provide a more realistic and moving experience by reflecting the user's emotions. Therefore, it is necessary to recognize the user's emotions and provide appropriate environmental settings and effects to enrich the recreation of memories in a virtual space.

[1411] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1412] In this invention, the server includes means for receiving uploaded image data and saving it in storage, means for passing the path of the saved image file to the generating AI, and means for saving the generated three-dimensional data and data reflecting emotional information in the virtual space platform. This makes it possible to generate and save a three-dimensional virtual space that takes emotional information into consideration based on image data taken by the user, and to provide the user with the experience of recreating moving memories.

[1413] "User terminal" refers to any electronic device that a user operates to send and receive data.

[1414] A "server" is a computer system that receives, stores, processes, and provides data over a network.

[1415] "Image data" refers to digital files that contain visual information, such as photographs and videos.

[1416] "Storage" refers to a physical or virtual medium for storing digital data.

[1417] "Generative AI" refers to artificial intelligence techniques for analyzing and generating data.

[1418] "Data for three-dimensional virtual space" refers to all digital information used to represent virtual space using three-dimensional computer graphics.

[1419] An "emotion engine" refers to technology that includes an algorithm for analyzing a user's emotions and generating an appropriate response based on that.

[1420] "Virtual space platform" refers to an integrated system of software and hardware that allows users to access and experience virtual spaces.

[1421] "Notification" refers to a message or alert that the system uses to communicate information to the user.

[1422] MODE FOR CARRYING OUT THE INVENTION

[1423] This invention is a system that recreates memories in a virtual space based on photos and videos saved by the user, and further enriches the recreation by recognizing the user's emotions. This system consists of a user terminal, a server, a generative artificial intelligence, an emotion engine, and a virtual space platform. The detailed processing of each hardware and software is described below.

[1424] Hardware and Software

[1425] User device: Refers to electronic devices such as smartphones, PCs, tablets, etc. Users select and upload image data using a dedicated application.

[1426] Server: A computer system that receives, stores, processes, and provides data. The server is equipped with storage and stores image data and generated 3D data.

[1427] Generative AI: AI technology that analyzes image data and generates 3D data. It extracts specific landscapes, people, and movement patterns to generate data for 3D virtual spaces.

[1428] Emotion engine: A technology that analyzes image data to recognize the user's emotions and generates environment settings and effects that reflect those emotions.

[1429] Virtual Space Platform: A system that allows users to access virtual spaces and relive memories. It can be accessed using a VR headset or a PC.

[1430] Program processing

[1431] Processing on the user's device

[1432] A user starts a dedicated application and selects and uploads photos and videos from their device. At this time, the device generates an upload request and sends the selected image data to the server. For example, a user selects and uploads several photos taken during a summer vacation with their family.

[1433] Processing on the server

[1434] The server receives the image data sent from the device and saves it in storage. Once the saving is complete, the server passes the file path to the AI ​​generator. For example, the server provides the AI ​​generator with the file path of a set of saved family trip photos.

[1435] Processing with generative artificial intelligence

[1436] The artificial intelligence begins analyzing the image data based on the file path received from the server. This allows it to extract the target scenery, people, and movement patterns, and generate data for a 3D virtual space. In this case, it recreates the beach scenery and family figures in 3D from photos of a family summer vacation.

[1437] Processing with the Emotion Engine

[1438] The emotion engine analyzes the image data and recognizes the user's emotions (e.g., joy, surprise, excitement). Based on this, the generative AI adjusts the environmental settings and effects of the virtual space. For example, if the user's emotion is recognized as "joy," the brightness and music in the virtual space will be adjusted.

[1439] Processing on a virtual space platform

[1440] The generated 3D data and the data reflecting the emotional information are sent back to the server and saved on the virtual space platform. Once saved, the server sends a notification to the user. The user can access the platform using the URL and login information in the notification and experience the memories recreated in the virtual space using a VR headset or PC.

[1441] Specific examples

[1442] Example of recreating family trip memories

[1443] An example of recreating a virtual space based on photos taken by a user on a summer vacation with their family would include the following steps:

[1444] 1. Upload

[1445] The user launches a dedicated app on their smartphone, selects a few photos from a family trip, and clicks the upload button.

[1446] 2. Data storage

[1447] The server receives the image data and stores it in storage.

[1448] 3. Image and Sentiment Analysis

[1449] Generative AI analyzes photos and an emotion engine recognizes emotions.

[1450] 4. Environment and Effects Settings

[1451] Adjusting environmental settings and effects in a virtual space based on emotional information.

[1452] 5. Data Retention and Notification

[1453] The server stores the generated data in the virtual space platform and notifies the user.

[1454] 6. Virtual Experience

[1455] Users use a VR headset to experience a virtual space that recreates memories of a family trip.

[1456] Prompt Sentence Examples

[1457] "Project: Memories of a summer family trip. Upload family photos on the screen. Then, we analyze emotions and add appropriate effects. For example, we create an environment that evokes feelings of joy or surprise."

[1458] This system allows users to recreate and experience memories more realistically based on the emotions they felt.

[1459] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1460] Step 1:

[1461] The user selects and uploads image data from the device using a dedicated application. Specifically, the user launches the smartphone app and clicks the "Upload Photos" button. They then select multiple photos from a family trip and press the upload button. The device generates an upload request based on this and sends the image data to the server.

[1462] Input: Image data selected by the user (photos, videos)

[1463] Output: Upload request to server

[1464] Step 2:

[1465] The server receives the image data sent from the terminal and stores it in storage. Specifically, the server stores the received image data file in a specified directory and registers the file path in the management system.

[1466] Input: Image data sent from the device

[1467] Output: Image file saved in storage, file path obtained

[1468] Step 3:

[1469] The server passes the file path of the stored image data to the generation AI. Specifically, the server provides the path of the image file as an input parameter of the generation AI and requests analysis.

[1470] Input: File path of saved image data

[1471] Output: Analysis request to the generative AI

[1472] Step 4:

[1473] Generative AI analyzes image data and extracts scenery, people, movement patterns, etc. For example, it can recognize scenery such as the sea and mountains, as well as family members' faces and movements from photos of a family trip, and use this information to generate data for a three-dimensional virtual space.

[1474] Input: Image data file path

[1475] Output: Data for 3D virtual space

[1476] Step 5:

[1477] The emotion engine analyzes image data and recognizes the user's emotions. Specifically, the emotion engine analyzes facial expressions and situations obtained from the photos to detect emotions such as joy and surprise. This emotional information is provided to the generative AI.

[1478] Input: Image data

[1479] Output: Recognized emotion information

[1480] Step 6:

[1481] The generative AI adjusts the environmental settings and effects in the virtual space based on the emotional information obtained from the emotion engine. Specifically, it selects the brightness and color of the lighting, background music, etc. based on the emotional information. For example, if joy is recognized, bright lighting and upbeat music will be set.

[1482] Input: Recognized emotion information

[1483] Output: Adjusted virtual space settings and effects

[1484] Step 7:

[1485] The server stores the data generated by the generative AI and emotion engine on the virtual space platform. Specifically, the server uploads the generated three-dimensional data and emotion information to the virtual space platform and stores them as data.

[1486] Input: 3D data, emotional information

[1487] Output: Data stored in the virtual space platform

[1488] Step 8:

[1489] The server notifies the user that the save has been completed. Specifically, the server sends a notification to the user's device that includes an access URL and login information.

[1490] Input: Completion information saved in the virtual space platform

[1491] Output: Notification to user (access URL, login information)

[1492] Step 9:

[1493] Based on the information provided, users can log in to the virtual space platform and experience the recreated memories. Specifically, users access the virtual space using a VR headset or PC and interact with the recreated scenery and people. They experience environmental settings and effects based on emotional information.

[1494] Input: Access URL, login information

[1495] Output: Experience in virtual space (recreated memory interaction)

[1496] (Application example 2)

[1497] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1498] Conventional virtual space reproduction systems based on photographs and videos have the problem that they are unable to create environmental settings and effects that take into account the user's emotions, resulting in a lack of realism in the experience. The present invention aims to solve this problem and provide a more emotionally rich and realistic virtual space experience.

[1499] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading image data selected by the user's terminal, means for the server to receive the uploaded image data and pass it to the generation AI, means for the generation AI to analyze the received image data and convert it into data for a three-dimensional virtual space, means for the emotion engine to analyze the image data and recognize the user's emotional information, means for the generation AI to adjust the environmental settings and effects of the virtual space based on the emotional information, means for the server to save the converted three-dimensional data in the virtual space platform, and means for the user to access the virtual space platform via their terminal and experience recreated memories. This enables a rich and realistic virtual space experience that reflects the user's emotions.

[1500] "User terminal" refers to a device used by a user to select and upload image data, and includes smartphones, tablets, PCs, etc.

[1501] "Image data" refers to data of photos and videos saved by the user, and is the basis for recreating the virtual space.

[1502] A "server" is a computer system that receives and stores uploaded image data, and also exchanges data with the generating artificial intelligence and virtual space platform.

[1503] "Generative AI" is an AI technology that analyzes image data and generates data for three-dimensional virtual space.

[1504] The "emotion engine" is a technology that recognizes the user's emotional information from analyzed image data and adjusts the environmental settings and effects of the virtual space based on that information.

[1505] A "virtual space platform" is a system that stores generated three-dimensional data and provides a virtual space for users to access and experience.

[1506] "Three-dimensional data" is data in a three-dimensional virtual space generated based on image data, and is data that represents objects and environments within the virtual space.

[1507] "Environmental settings" are settings that adjust environmental elements such as lighting and music within the virtual space.

[1508] "Effects" are visual and tactile effects that enrich the user's experience within the virtual space.

[1509]

[1510] The present invention is a system that recreates memories in a virtual space based on photos and videos saved by the user, and further enriches the recreation by recognizing the user's emotions. This system includes a user terminal, a server, a generative artificial intelligence, an emotion engine, and a virtual space platform.

[1511] First, the user's device has the function to upload selected image data (photos and videos). The user selects and uploads image data from their device through a dedicated application. An upload request is generated and the image data is sent to the server.

[1512] The server saves the received image data in storage. Once saving is complete, it passes the file path to the generating AI. The generating AI begins analyzing the image data and extracts the target scenery, people, movement patterns, etc. Based on this, data for the three-dimensional virtual space is generated.

[1513] Next, the emotion engine analyzes the image data and recognizes the user's emotions. Based on this recognized emotion information, the generative AI makes further fine adjustments. Specifically, it adjusts the environmental settings (e.g., lighting and music) and effects (visual effects such as warmth and coolness) in the virtual space based on the emotion information.

[1514] The generated 3D data and the data reflecting the emotional information are returned to the server and saved on the virtual space platform. After the server confirms that saving is complete, it sends a notification to the user. This notification includes the URL and login information for accessing the virtual space platform.

[1515] Users log in to the virtual space platform and access the recreated space using the URL they receive. Using a VR headset or PC, users can walk around the virtual space and interact with the scenery and people from their memories. Furthermore, the recreated space has environmental settings and effects that are adjusted based on the user's emotions, making for a more realistic and emotional experience.

[1516] Specific examples

[1517] Example of recreating family trip memories

[1518] 1. Upload:

[1519] A user launches the application on their smartphone, clicks the "Upload Photos" button in the application, selects a few photos from a family trip, and uploads them to the server using a dedicated function in the application.

[1520] 2. Data storage:

[1521] The server receives the uploaded photo and saves it in storage. Once the saving is complete, the file path is passed to the AI ​​generator.

[1522] 3. Image and Sentiment Analysis:

[1523] The generation AI receives the file path and begins image analysis. The system extracts images of family members and travel destinations (such as the ocean or mountains) and generates data for a 3D virtual space based on this. At the same time, the emotion engine analyzes the image data and recognizes the user's emotions (e.g., joy, surprise, excitement).

[1524] 4. Environment and Effects Settings:

[1525] Based on the emotional information recognized by the emotion engine, the generative AI adjusts the environmental settings (e.g., music and lighting) and effects (e.g., specific visual effects) in the virtual space, thereby reinforcing the emotions felt by the user at the time.

[1526] 5. Data Retention and Notification:

[1527] The server receives the generated 3D data and the data reflecting the emotional information, uploads it to the virtual space platform, and saves it. After saving is complete, the server notifies the user of the completion of saving and provides access information.

[1528] 6. Virtual Experience:

[1529] Users log in to the virtual space platform using the provided access information. They then put on a VR headset and can relive their family trip memories in the virtual space created. They can walk around, interact with the recreated scenery and people, and enjoy the environmental settings and effects that recreate the emotions of the time.

[1530] Prompt Sentence Examples

[1531] "I uploaded three photos and one video of my family at a carnival. Use these to create a virtual space filled with fun and excitement. Add colorful lights and lively music."

[1532] This system enables a rich and realistic virtual space experience that reflects the user's emotions.

[1533] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1534] Step 1:

[1535] The user launches the application on their smartphone and clicks the dedicated "Photo Upload" button. The user selects several photos and videos from a family trip from within the application and generates an upload request. At this time, the selected image data is imported into the application.

[1536] Step 2:

[1537] The user's device sends the generated upload request to the server. The server receives this request and saves the image data to storage. Once saving is complete, the file path is recorded. The image data received as input is saved and a file path is generated as output.

[1538] Step 3:

[1539] The server passes the saved file path to the generation AI. The generation AI receives the file path and begins analyzing the image data. It extracts the target scenery, people, and movement patterns from the image data, and generates data for a three-dimensional virtual space based on that. It receives image data as input and generates three-dimensional data as output.

[1540] Step 4:

[1541] The emotion engine analyzes the image data received from the generation AI and recognizes the user's emotional information (e.g., enjoyment, surprise, excitement). Based on the recognized emotional information, the generation AI adjusts the environmental settings (e.g., lighting and music) and effects (visual effects such as warmth and coolness) of the virtual space. It receives the user's emotional information as input and generates adjusted three-dimensional data as output.

[1542] Step 5:

[1543] The generated 3D data and the data reflecting the emotion information are returned to the server. The server uploads and saves this data to the virtual space platform. After confirming that saving is complete, the server sends a notification to the user. This notification includes the URL and login information for accessing the virtual space platform. It receives 3D data as input and generates a user notification as output.

[1544] Step 6:

[1545] The user logs into the virtual space platform using the notified URL and login information. Using a VR headset or PC, the user relives the memories of their family trip in the created virtual space. The user can walk around the virtual space and interact with the recreated scenery and people. The recreated space has environmental settings and effects adjusted based on emotions, making for a more realistic and moving experience. The notified information is used as input, and access to and experience of the virtual space is obtained as output.

[1546] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1547] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1548] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1549] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1550] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1551] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1552] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1553] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1554] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1555] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1556] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1557] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1558] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1559] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1560] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1561] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1562] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1563] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1564] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1565] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1566] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1567] The following is further disclosed regarding the above embodiment.

[1568] (Claim 1)

[1569] A means for uploading the selected image data to the user's terminal;

[1570] A means for the server to receive the uploaded image data and pass it to the generating artificial intelligence;

[1571] A means for analyzing the image data received by the generating artificial intelligence and converting it into data for a three-dimensional virtual space;

[1572] A means for the server to store the converted three-dimensional data in a virtual space platform;

[1573] The system includes a means for a user to access the virtual space platform via a terminal and experience recreated memories.

[1574] (Claim 2)

[1575] 10. The system of claim 1, further comprising means for notifying a user of the three-dimensional data stored in the virtual space platform.

[1576] (Claim 3)

[1577] 2. The system according to claim 1, further comprising means for reproducing emotions and movements at a time selected by the user based on the analyzed image data.

[1578] "Example 1"

[1579] (Claim 1)

[1580] means for uploading the image data selected by the user's information processing device;

[1581] A means for the central processing unit to receive the uploaded image data and pass it to the generating artificial intelligence;

[1582] A means for analyzing the image data received by the generating artificial intelligence and converting it into data for a three-dimensional virtual environment;

[1583] a means for the central processing unit to store the converted three-dimensional data in a virtual environment platform;

[1584] The system includes a means for a user to access the virtual environment platform via an information processing device and experience recreated memories.

[1585] (Claim 2)

[1586] 10. The system of claim 1, further comprising means for notifying a user of the three-dimensional data stored in the virtual environment platform.

[1587] (Claim 3)

[1588] 2. The system according to claim 1, further comprising means for reproducing emotions and movements at a time selected by the user based on the analyzed image data.

[1589] "Application Example 1"

[1590] (Claim 1)

[1591] A means for uploading selected media data to a user terminal;

[1592] A means for the server to receive the uploaded media data and pass it to the generating artificial intelligence;

[1593] A means for analyzing the media data received by the generating artificial intelligence and converting it into data for a three-dimensional virtual space;

[1594] A means for the server to store the converted three-dimensional data in a virtual space platform;

[1595] A means for users to access the virtual space platform via a terminal and experience the recreated memories;

[1596] A system that includes a means for a user to move and interact within the generated three-dimensional space.

[1597] (Claim 2)

[1598] 10. The system of claim 1, further comprising means for notifying a user of the three-dimensional data stored in the virtual space platform.

[1599] (Claim 3)

[1600] 10. The system of claim 1, further comprising means for reproducing emotions and movements at a time selected by the user based on the analyzed media data.

[1601] "Example 2: Combining Emotion Engines"

[1602] (Claim 1)

[1603] A means for uploading the selected image data to the user's terminal;

[1604] A means for the server to receive the uploaded image data and store it in storage;

[1605] A means for the server to pass the path of the saved image file to the generated artificial intelligence;

[1606] A means for analyzing the image data received by the generating artificial intelligence and converting it into data for a three-dimensional virtual space;

[1607] a means for the emotion engine to analyze the image data and provide the recognized emotion information to the generating artificial intelligence;

[1608] A means for the generative AI to adjust the environmental settings and effects in the virtual space based on the emotional information;

[1609] a means for storing the generated three-dimensional data and the data reflecting the emotional information in a virtual space platform;

[1610] The system includes a means for a user to access the virtual space platform via a terminal and experience recreated memories.

[1611] (Claim 2)

[1612] 10. The system of claim 1, further comprising means for notifying a user of the three-dimensional data stored in the virtual space platform.

[1613] (Claim 3)

[1614] 2. The system according to claim 1, further comprising means for reproducing emotions and movements at a time selected by the user based on the analyzed image data.

[1615] "Application example 2 when combining emotion engines"

[1616] (Claim 1)

[1617] A means for uploading the selected image data to the user's terminal;

[1618] A means for the server to receive the uploaded image data and pass it to the generating artificial intelligence;

[1619] A means for analyzing the image data received by the generating artificial intelligence and converting it into data for a three-dimensional virtual space;

[1620] A means for the emotion engine to analyze image data and recognize emotion information of a user;

[1621] A means for the generative AI to adjust the environmental settings and effects of the virtual space based on emotional information;

[1622] A means for the server to store the converted three-dimensional data in a virtual space platform;

[1623] The system includes a means for a user to access the virtual space platform via a terminal and experience recreated memories.

[1624] (Claim 2)

[1625] 10. The system of claim 1, further comprising means for notifying a user of the three-dimensional data stored in the virtual space platform.

[1626] (Claim 3)

[1627] 2. The system according to claim 1, further comprising means for reproducing emotions and movements at a time selected by the user based on the analyzed image data and emotion information. [Explanation of symbols]

[1628] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for uploading the selected image data to the user's terminal; A means for the server to receive the uploaded image data and pass it to the generating artificial intelligence; A means for analyzing the image data received by the generating artificial intelligence and converting it into data for a three-dimensional virtual space; A means for the server to store the converted three-dimensional data in a virtual space platform; The system includes a means for a user to access the virtual space platform via a terminal and experience recreated memories.

2. 10. The system of claim 1, further comprising means for notifying a user of the three-dimensional data stored in the virtual space platform.

3. 2. The system according to claim 1, further comprising means for reproducing emotions and movements at a time selected by the user based on the analyzed image data.

Images