system

Abstract

To provide a system that allows multiple users to share the experience of the same virtual space. [Solution] A system including a means for a user to upload 2D data from a terminal, a means for receiving the 2D data and applying an artificial intelligence model to convert it into 3D data, a means for placing the converted 3D data in a virtual space, a means for a user to experience the 3D data in the virtual space via a virtual reality device, and a means for multiple users to share and experience the 3D data in the same virtual space.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional albums and photos are limited to two-dimensional surfaces, which can lack a sense of realism when reminiscing about memories. Furthermore, when sharing photos with others, there are many visual constraints, making it difficult to share the experience. Furthermore, there are limited ways for people who are far apart to experience the same memories in real time, resulting in a lack of a sense of shared memory. The goal is to solve these problems and provide a more vivid and shared memory experience. [Means for solving the problem]

[0005] The present invention is a system including a means for a user to upload two-dimensional data from a terminal, a means for receiving the two-dimensional data and applying an artificial intelligence model to convert it into three-dimensional data, a means for placing the converted three-dimensional data in a virtual space, a means for a user to experience the three-dimensional data in the virtual space via a virtual reality device, and a means for multiple users to share and experience the three-dimensional data in the same virtual space.

[0006] According to the present invention, realistic 3D data is reconstructed by analyzing uploaded 2D data, identifying the shape and depth of people and objects, generating a 3D mesh based on the identified shape and depth, and applying texture to the generated 3D mesh and mapping color information extracted from the original 2D data. Furthermore, a shared experience can be provided by synchronizing the location information and viewpoint data of multiple users in real time, displaying avatars of other users, and receiving invitation requests from users and granting viewing privileges to other users. In this way, a sense of realism and sharing can be achieved, exceeding the limitations of conventional 2D photographs.

[0007] "User" refers to the entity that uses the system to upload 2D data and view and share 3D data in the virtual space.

[0008] A "terminal" is a device operated by a user and used to upload two-dimensional data and view virtual space.

[0009] "Two-dimensional data" refers to information such as images and photographs that exist on a plane.

[0010] "3D data" is digital information that has a three-dimensional shape and depth, and is used for display in virtual space.

[0011] "Artificial intelligence model" refers to machine learning algorithms and neural networks that analyze two-dimensional data and convert it into three-dimensional data.

[0012] A "virtual space" is a three-dimensional digital environment generated using computer technology that can be experienced by a user through a virtual reality device.

[0013] "Virtual reality devices" are devices used to experience virtual spaces, and specifically refer to head-mounted displays and VR goggles.

[0014] A "mesh" is a network of polygons used to represent a three-dimensional structure in three-dimensional data.

[0015] A "texture" is detailed surface color and pattern information applied to a 3D mesh.

[0016] An "avatar" refers to a digital character or icon that represents a user in a virtual space.

[0017] "Viewpoint data" is information that indicates the direction in which the user is looking within the virtual space.

[0018] "Viewing authority" means granting a specific user the right to view three-dimensional data.

[0019] An "invitation request" is a request sent to the system to invite another user to a virtual space.

[0020] "Real-time" refers to processing and synchronization occurring in real time. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0022] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0025] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0026] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0027] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0029] [First embodiment]

[0030] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0031] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0034] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0037] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0041] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0042] The present invention is a system that allows users to experience 2D data in 3D in a virtual space and share that experience among multiple users. This system mainly consists of the following steps.

[0043] 1. The user uploads 2D data

[0044] Users use their own devices to select and upload 2D data such as photos and images from their local disks, which the devices then receive and send to the server.

[0045] 2. The server receives the 2D data and sends it to the AI ​​model.

[0046] The server receives the uploaded 2D data, verifies the data format and file size, and then sends it to an AI model, which uses machine learning algorithms to analyze the 2D data and convert it into 3D data.

[0047] 3. Artificial intelligence models transform 2D data into 3D

[0048] The AI ​​model uses image analysis technology to identify the shape and depth of people and objects from 2D data. Based on this information, it generates a 3D mesh and applies textures to it, resulting in realistic 3D data.

[0049] 4. The server places the 3D data in the virtual space

[0050] The generated 3D data is placed in a virtual space by the server, and the space is ready for the user to view using a virtual reality device.

[0051] 5. The user experiences the virtual space using a virtual reality device.

[0052] Users wear virtual reality devices and launch a dedicated application to view and experience 3D data placed in a virtual space. Users can change their viewpoint and walk around the virtual space.

[0053] 6. Shared experiences among multiple users

[0054] Users can invite other users to the virtual space. Invited users can also use the same virtual reality device and application to experience the same virtual space. The server synchronizes the location and viewpoint data of multiple users in real time and displays each user's avatar, enabling a shared experience.

[0055] Specific examples

[0056] As a specific example, a case where photos of a family trip are used will be described.

[0057] User A: Uploads a group photo of a family trip from the device. The photo is a file called "familytrip.jpg." User A operates the device, selects the photo, and clicks the "Upload" button.

[0058] Server: Receives the upload request, saves "Family Trip.jpg", and sends it to the AI ​​model, which analyzes the photo, identifies the positions of family faces and the background scenery, and generates a 3D mesh.

[0059] Artificial intelligence model: Generates a 3D mesh and applies texture based on color information extracted from the original photo.

[0060] Server: Uploads the generated 3D data to the virtual space and links it to User A's profile.

[0061] User A: Puts on the VR goggles, launches the Metaverse application, and selects the "Family Trip" album. User A relives the family trip in a virtual space in a realistic way.

[0062] User A: Furthermore, User B invites Friend B to this virtual space. The server grants Friend B permission to view the space, and Friend B also puts on VR goggles and joins the virtual space. The two can relive memories of their family trip together and communicate in real time.

[0063] This system allows users to not only memorize memorable scenes as photographs, but also recreate them in three dimensions, enabling shared experiences in real time.

[0064] The processing flow will be explained below.

[0065] Step 1:

[0066] The user uses the terminal to select two-dimensional data (e.g., a photo file).

[0067] The user opens a browser or a dedicated application, accesses the photo upload screen, selects a file such as "Family Trip.jpg" from the local disk, and clicks the upload button.

[0068] Step 2:

[0069] The device sends the selected two-dimensional data to the server.

[0070] The device generates an HTTP POST request and sends the selected photo data and user ID to the server, along with metadata such as file name, file format, and file size.

[0071] Step 3:

[0072] The server receives and verifies the 2D data.

[0073] The server receives the HTTP POST request, saves the photo data in temporary storage, and performs validation checks such as checking for valid file formats and file sizes, and virus checks.

[0074] Step 4:

[0075] The server sends the photo data to an artificial intelligence model.

[0076] Once verified, the photo data is passed to the AI ​​model, which then sends the photo data and associated metadata to the AI ​​model.

[0077] Step 5:

[0078] An artificial intelligence model analyzes the two-dimensional data.

[0079] The AI ​​model identifies people and objects in the photo, analyzes their shape and depth, and uses deep learning algorithms to analyze each pixel in the photo and generate mesh data to recreate the three-dimensional structure.

[0080] Step 6:

[0081] An artificial intelligence model generates 3D meshes and textures.

[0082] A three-dimensional shape is created based on the mesh data. Color information is extracted from the original photo and texture is applied to the generated mesh. This results in a 3D model that looks realistic.

[0083] Step 7:

[0084] The server places the generated 3D data in a virtual space.

[0085] The completed 3D data is placed at a specific location in the virtual space. If multiple 3D objects are required, each object is placed in the appropriate location.

[0086] Step 8:

[0087] The server links the 3D data to the user's profile.

[0088] The generated 3D data ID is linked to the user ID and saved in a database, allowing the user to access the 3D data at any time.

[0089] Step 9:

[0090] The user puts on the virtual reality device and starts a dedicated application.

[0091] The user puts on the VR goggles and starts the Metaverse application. After the user is authenticated, a list of available albums is displayed within the application.

[0092] Step 10:

[0093] The user selects and views the generated 3D album.

[0094] Select "Family Trip" from the album list and view the 3D data placed in the virtual space. Users can change the viewpoint and walk around the virtual space.

[0095] Step 11:

[0096] A user sends a request to the server to invite others.

[0097] A user generates a request on their device to invite other users to the same virtual space and sends it to the server, which generates an invitation link or sharing code.

[0098] Step 12:

[0099] The server adds viewing permissions to other people's profiles.

[0100] The server receives the invitation request and adds the permission to view the three-dimensional data to the profile of the other specified user.

[0101] Step 13:

[0102] Other users access the virtual space using the invitation link.

[0103] Other users click on the invitation link and join the same virtual space using their virtual reality devices. The server synchronizes data in real time, and multiple avatars are displayed simultaneously.

[0104] Step 14:

[0105] The server synchronizes the location information and viewpoint data among multiple users.

[0106] The server receives each user's position and viewpoint data in real time and transmits that information to other users, allowing each user to see the avatars of other users in real time.

[0107] This series of steps allows users to re-experience a simple two-dimensional photo in a realistic three-dimensional space and share it with other users.

[0108] Example 1

[0109] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0110] Modern virtual reality technology is often experienced by users alone, and individual experiences tend to be fragmented. Furthermore, the process of converting 2D data into 3D and placing it in a virtual space is extremely complex and requires extensive specialized knowledge. As a result, it is difficult for ordinary users to create realistic 3D experiences from 2D data and share them with other users. Furthermore, it is difficult for multiple users to share an experience in real time in the same virtual space, limiting collaborative experiences.

[0111] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0112] In this invention, the server includes a means for users to upload 2D data from their devices, a means for receiving the 2D data and applying a generative AI model to convert it into 3D data, and a means for placing the converted 3D data in a virtual space. This allows general users to easily convert 2D data into 3D data, create realistic virtual experiences, and share them with other users. It also allows multiple users to experience 3D data in real time within the same virtual space, synchronizing each user's location information and viewpoint data. As a result, users can enjoy a collaborative virtual experience, enabling virtual reality technology to be enjoyed on a wider scale.

[0113] A "user" is a person who uses a terminal to upload two-dimensional data and experience virtual reality.

[0114] A "terminal" refers to an electronic device used by a user, such as a computer or smartphone.

[0115] "Two-dimensional data" refers to two-dimensional information such as photos and image data uploaded by users.

[0116] "Upload" refers to the act of a user sending data from a terminal to a server.

[0117] A "server" is a computer system that processes two-dimensional data received from users, converts it into three-dimensional data, and places it in a virtual space.

[0118] A "generative AI model" is a program that uses artificial intelligence technology to analyze two-dimensional data and convert it into three-dimensional data.

[0119] "3D data" refers to three-dimensional data converted from 2D data by a generative AI model.

[0120] "Virtual space" refers to a three-dimensional virtual space generated by computer graphics.

[0121] A "virtual reality device" refers to a head-mounted display or VR goggles that users use to experience a virtual space.

[0122] A "shared experience" is when multiple users simultaneously experience virtual reality in the same virtual space.

[0123] "Location information" refers to coordinate data within the user's virtual space.

[0124] "Viewpoint data" refers to data that indicates the visual direction and angle of the user within the virtual space.

[0125] An "invitation request" is a request made by a user to invite another user to a virtual space.

[0126] "Viewing authority" refers to the authority that allows an invited user to view data in a virtual space.

[0127] An "avatar" is a three-dimensional character that represents a user in a virtual space.

[0128] "Real-time synchronization" refers to the simultaneous updating of location information and viewpoint data for multiple users within a virtual space.

[0129] The present invention is a system that allows users to upload 2D data from their devices, convert it into 3D data, and share it in a virtual space. The steps for implementing this system are as follows:

[0130] First, the user uploads 2D data using a terminal. The terminal can be an electronic device such as a PC or smartphone. The user opens a file selection dialog, selects a photo or image data, such as "Family Trip.jpg," from the local disk, and clicks the upload button to send the data to the server. The technology used is an HTTP POST request.

[0131] Next, the server receives the 2D data. Using a web framework such as Flask or Django, the server stores the data appropriately and validates the data format and file size. The server then sends the 2D data to the artificial intelligence model for generative AI model application. The server uses a REST API or JSON format for this purpose.

[0132] The generative AI model uses machine learning libraries such as TENSORFLOW® and PyTorch to analyze 2D data and convert it into 3D data. The AI ​​model analyzes the received 2D data, extracting object contours and estimating depth. Based on the results of this analysis, a 3D mesh is generated and textures are applied using color information extracted from the original image.

[0133] The server receives the generated 3D data and places it in a virtual space. The virtual space is built using a virtual space engine such as Unity or Unreal Engine. The server stores the generated 3D data in a database and places it appropriately in the virtual space.

[0134] Users wear virtual reality devices (such as Oculus Rift or HTC Vive) and view and experience 3D data in a virtual space. By launching a dedicated application, users can move freely within the virtual space and interact with it.

[0135] In addition, multiple users can share the experience of 3D data in the same virtual space. Users can invite other users to the virtual space, and the invited users also use virtual reality devices. The server synchronizes the location and viewpoint data of multiple users in real time and displays the avatars of other users.

[0136] Specific examples

[0137] As a concrete example, let's consider the case of a family trip photo. User A uploads a group photo of the family trip, "familytrip.jpg," from their device. The server receives the upload request, saves "familytrip.jpg," and sends it to the generative AI model. The generative AI model analyzes the photo, identifies the position of family members' faces and the background scenery, and generates a 3D mesh. The server uploads the generated 3D data to the virtual space and links it to User A's profile. User A puts on VR goggles, launches the Metaverse application, and selects the "Family Trip" album. Within the virtual space, they relive their family trip memories in a way that is close to reality. User A invites Friend B to this virtual space, and the server grants Friend B viewing permissions as well. Friend B also puts on VR goggles and joins the virtual space, and the two can experience their family trip memories together and communicate in real time.

[0138] Prompt Sentence Examples

[0139] What are the steps to upload photos from a family trip, turn them into 3D, experience them in a virtual space using a VR device, and share them with friends?

[0140] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0141] Step 1:

[0142] The user selects an image on the device

[0143] The user opens a file selection dialog using the terminal and selects an image to upload from the local disk (for example, "Family Trip.jpg"). The input is the 2D data (image file) selected by the user. The output is the path information of the selected image file.

[0144] Step 2:

[0145] The device sends the data to the server

[0146] When the user clicks the "Upload" button, the device sends the selected image data to the server as an HTTP POST request. Specifically, the image data is sent as form data using the JavaScript (registered trademark) fetch API. The input is the path information of the selected image file, and the output is the image data sent to the server.

[0147] Step 3:

[0148] The server receives the data

[0149] The server receives the image data sent from the device. It processes the HTTP POST request using a web framework such as Flask or Django. Specifically, the server temporarily stores the image data and verifies the format and size. The input is the image data in the HTTP request, and the output is the verified image data.

[0150] Step 4:

[0151] The server sends the data to the artificial intelligence model

[0152] The server sends the received image data to the generative AI model. Specifically, it uses a REST API to send the image data in JSON format. The input is the verified image data, and the output is an analysis request to the artificial intelligence model.

[0153] Step 5:

[0154] An artificial intelligence model analyzes the data

[0155] The generative AI model uses TensorFlow and PyTorch to perform image analysis. Specifically, it performs object recognition and depth estimation in the image to generate a 3D mesh. The input is 2D image data sent from the server, and the output is 3D mesh data.

[0156] Step 6:

[0157] Applying textures to 3D meshes

[0158] The AI ​​model applies texture to the generated 3D mesh using color information extracted from the original image. Specifically, it uses texture mapping technology. The input is 3D mesh data and texture information, and the output is the completed 3D data.

[0159] Step 7:

[0160] The server places the 3D data in the virtual space

[0161] The server receives the generated 3D data and places it in a virtual space created with Unity or Unreal Engine. Specifically, it stores the data in a database and transfers it to the virtual space engine. The input is the completed 3D data, and the output is 3D data placed in the virtual space.

[0162] Step 8:

[0163] Users experience a virtual space using a virtual reality device.

[0164] The user puts on the VR goggles and launches a dedicated application. Specific actions include visually viewing and manipulating objects in a virtual space. The input is 3D data placed in the virtual space, and the output is the visual information and interaction information of the virtual reality that the user experiences.

[0165] Step 9:

[0166] Shared experiences among multiple users

[0167] Users can invite other users into the virtual space, and the server synchronizes each user's location information and viewpoint data in real time. Specifically, the invited users also wear the same virtual reality device and use real-time communication technology. The input is the location information and viewpoint data of multiple users, and the output is a real-time shared virtual space experience.

[0168] Prompt Sentence Examples

[0169] What are the steps to upload photos from a family trip, turn them into 3D, experience them in a virtual space using a VR device, and share them with friends?

[0170] (Application example 1)

[0171] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0172] The present invention relates to a system that not only allows users to experience two-dimensional data in three dimensions within a virtual space, but also allows multiple users to share that experience in real time. In particular, the objective of the present invention is to provide a system that allows users to enjoy product experiences in a virtual store in three dimensions and share shopping experiences with other users.

[0173] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0174] In this invention, the server includes means for users to upload 2D data from their terminals, means for receiving the 2D data and applying an artificial intelligence model to convert it into 3D data, means for arranging the converted 3D data in a virtual space, means for users to experience the 3D data in the virtual space through a virtual reality device, means for multiple users to share the experience of the 3D data in the same virtual space, means for users to upload product images and view and experience products in a 3D manner in a virtual store, and means for inviting other users to the virtual store and sharing the shopping experience in real time, thereby enabling users to experience products in a 3D manner in the virtual store and share that experience with other users.

[0175] A "terminal" refers to an electronic device used by a user, such as a computer, smartphone, or tablet.

[0176] "Two-dimensional data" refers to digital data that contains two-dimensional information in terms of length and width, such as images and photographs.

[0177] "Upload" refers to the action of sending data from a user's terminal to a server.

[0178] "3D data" refers to three-dimensional digital data that has depth.

[0179] An "artificial intelligence model" is a data analysis technology that uses machine learning algorithms and has the ability to convert two-dimensional data into three-dimensional data.

[0180] A "virtual space" is a virtual environment that differs from reality and is constructed on a computer using digital technology.

[0181] A "virtual reality device" is a device that allows users to have an experience similar to the real world in a digital space, such as VR goggles or a head-mounted display.

[0182] "Product images" refer to photographs or digital images of products used in online shopping, etc.

[0183] A "virtual store" is a virtual shop that exists on the Internet where you can browse and purchase products.

[0184] "Viewing in 3D" refers to the act of using 3D data to display depth information, providing an experience that feels as if you are looking at the real thing.

[0185] A "shopping experience" is a series of online activities in which a user browses products and considers a purchase.

[0186] A "shared experience" refers to multiple users experiencing the same experience in real time in the same virtual space.

[0187] "Real-time" refers to processing and information exchange occurring immediately, without delay.

[0188] System Overview

[0189] The present invention relates to a system that allows users to experience two-dimensional data in a three-dimensional virtual space and share that experience among multiple users. In particular, the present invention provides a system that allows users to view products in a virtual store in three dimensions and share their shopping experience with other users.

[0190] Hardware and Software Used

[0191] Server: Use a high-performance server, e.g. AWS EC2.

[0192] Virtual reality devices: Users use VR goggles or head-mounted displays. Example: Oculus Rift.

[0193] Devices: Smartphones, smart glasses, head-mounted displays, etc. Examples: Google® Glass®, HoloLens®.

[0194] Machine learning libraries: TensorFlow, PyTorch.

[0195] 3D modeling software: Blender, Unity engine.

[0196] Details of data processing and calculation

[0197] 1. Data upload

[0198] The user uploads a product image using their device. For example, they send "New Product Bag.jpg" from their smartphone to the server. The API used for this is the REST API.

[0199] 2. Image analysis and 3D conversion

[0200] The server analyzes the received product images using machine learning algorithms. It uses a convolutional neural network (CNN)-based model to identify the shape and depth of the object in the image, then generates a 3D mesh and applies textures to it.

[0201] 3. Placement in virtual space

[0202] The server places the generated 3D data in the virtual store using the Unity engine, which optimally positions the 3D model in the virtual space.

[0203] 4. Virtual Experiences

[0204] Users wear VR goggles and experience a virtual store through the application. The server synchronizes the user's viewpoint data and location information in real time, allowing users to freely walk around the virtual store and experience the products in three dimensions.

[0205] 5. Shared Experiences

[0206] Users can invite other users to the virtual store. The server receives the invitation request and grants viewing privileges to other users. Location and viewpoint data are synchronized in real time between multiple users, allowing them to share their shopping experience.

[0207] Specific examples

[0208] User A uploads "New Product Bag.jpg" from their smartphone.

[0209] After receiving the image, the server converts it into a 3D bag using an artificial intelligence model and places it inside the virtual store.

[0210] User A puts on VR goggles and experiences the sensation of actually picking up a bag in a virtual store.

[0211] User A invites friend B to a virtual store and they shop together while viewing new products in real time.

[0212] Prompt Sentence Examples

[0213] Design an AI model that generates a 3D model from an image. The input is a product image file (e.g., "New Product Bag.jpg"), and the output is a 3D mesh and texture for display in a virtual space. The model must identify shape and depth from the image and generate a realistic 3D model. The 3D model must also be optimized for effective display in a virtual store.

[0214] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0215] Step 1:

[0216] The user uploads 2D data from their device. The input is a product image (e.g., "New Product Bag.jpg"), and the output is the image sent to the server. This data is sent to the server and received using a REST API.

[0217] Step 2:

[0218] The server validates the received 2D data. The input is the uploaded image file, and the output is the validation result to see if the data format and file size are correct. If the data is not in the correct format at this step, an error message is returned.

[0219] Step 3:

[0220] The server sends the data to an artificial intelligence model, which converts it into 3D data. The input is verified 2D data, and the output is 3D data with 3D mesh and texture. The model used here is a Convolutional Neural Network (CNN)-based analysis algorithm that identifies shape and depth to generate a 3D model.

[0221] Step 4:

[0222] The server places the generated 3D data in a virtual space. The input is 3D data, and the output is 3D data placed at specific coordinates in the virtual space. The Unity engine is used to render the 3D model in the virtual space.

[0223] Step 5:

[0224] Users wear virtual reality devices and experience 3D data in a virtual space. The input is the user's viewpoint data and position information, and the output is an interactive experience in the virtual space. The server synchronizes this data in real time, allowing users to move freely within the virtual space.

[0225] Step 6:

[0226] Users can invite other users to a virtual space to share experiences. The input is an invitation request and the other user's ID, and the output is the granting of viewing permissions to the other users. Upon receiving the invitation, the server synchronizes the location and viewpoint data of multiple users in real time, providing a shared experience in the virtual space.

[0227] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0228] This invention is a system that allows users to experience 2D data in 3D in a virtual space and share that experience with multiple users, and also adds technology that recognizes the user's emotions and dynamically changes the objects and effects in the virtual space according to the emotions. This system mainly consists of the following steps.

[0229] 1. The user uploads 2D data

[0230] Users use their own devices to select and upload 2D data such as photos and images from their local disks, which the devices then receive and send to the server.

[0231] 2. The server receives the 2D data and sends it to the AI ​​model.

[0232] The server receives the uploaded 2D data, verifies the data format and file size, and then sends it to an AI model, which uses machine learning algorithms to analyze the 2D data and convert it into 3D data.

[0233] 3. Artificial intelligence models transform 2D data into 3D

[0234] The AI ​​model uses image analysis technology to identify the shape and depth of people and objects from 2D data. Based on this information, it generates a 3D mesh and applies textures to it, resulting in realistic 3D data.

[0235] 4. The server places the 3D data in the virtual space

[0236] The generated 3D data is placed in a virtual space by the server, and the space is ready for the user to view using a virtual reality device.

[0237] 5. The user experiences the virtual space using a virtual reality device.

[0238] Users wear virtual reality devices and launch a dedicated application to view and experience 3D data placed in a virtual space. Users can change their viewpoint and walk around the virtual space.

[0239] 6. Shared experiences among multiple users

[0240] Users can invite other users to the virtual space. Invited users can also use the same virtual reality device and application to experience the same virtual space. The server synchronizes the location and viewpoint data of multiple users in real time and displays each user's avatar, enabling a shared experience.

[0241] 7. Introducing an emotion recognition engine

[0242] The user's facial expression is captured using a camera and microphone, and the captured facial expressions and vocal tone are sent to an emotion recognition engine, which analyzes the data and determines the user's emotional state in real time.

[0243] 8. Changes in virtual space based on emotional data

[0244] Based on the recognized emotion data, objects and effects in the virtual space are dynamically changed. For example, if the user makes a surprised expression, all the objects in the virtual space will move in unison, or special effects will be displayed.

[0245] 9. Emotion Sharing among Multiple Users

[0246] When multiple users are in the same virtual space, they share their emotional data in real time. The server receives this emotional data and reflects it on each user's avatar and environment, making it easier for other users to see their emotional state.

[0247] Specific examples

[0248] As a specific example, a case where photos of a family trip are used will be described.

[0249] User A: Uploads a group photo of a family trip from the device. The photo is a file called "familytrip.jpg." User A operates the device, selects the photo, and clicks the "Upload" button.

[0250] Server: Receives the upload request, saves "Family Trip.jpg", and sends it to the AI ​​model, which analyzes the photo, identifies the positions of family faces and the background scenery, and generates a 3D mesh.

[0251] Artificial intelligence model: Generates a 3D mesh and applies texture based on color information extracted from the original photo.

[0252] Server: Uploads the generated 3D data to the virtual space and links it to User A's profile.

[0253] User A: Puts on the VR goggles, launches the Metaverse application, and selects the "Family Trip" album. User A relives the family trip in a virtual space in a realistic way.

[0254] Emotion recognition engine: Monitors user A's facial expressions and voice and analyzes their emotional state in real time. For example, if user A smiles, a character in the virtual space may greet them.

[0255] User A: Then, he invites Friend B to join the virtual space. The server grants Friend B permission to view the content, and Friend B also puts on VR goggles and joins the virtual space. The two can relive memories of their family trip together and share their emotions in real time. For example, if User A makes a surprised expression, that facial expression data is transmitted to Friend B, and a surprise effect may be displayed in the virtual space.

[0256] This system allows users to not only memorize memorable scenes as two-dimensional photographs, but also recreate them in three dimensions, allowing them to be experienced and shared in a virtual space that changes dynamically according to the user's emotions.

[0257] The processing flow will be explained below.

[0258] Step 1:

[0259] The user uses the terminal to select two-dimensional data (e.g., a photo file).

[0260] The user opens a browser or a dedicated application, accesses the photo upload screen, selects a file such as "Family Trip.jpg" from the local disk, and clicks the upload button.

[0261] Step 2:

[0262] The device sends the selected two-dimensional data to the server.

[0263] The device generates an HTTP POST request and sends the selected photo data and user ID to the server, along with metadata such as file name, file format, and file size.

[0264] Step 3:

[0265] The server receives and verifies the 2D data.

[0266] The server receives the HTTP POST request, saves the photo data in temporary storage, and performs validation checks such as checking for valid file formats and file sizes, and virus checks.

[0267] Step 4:

[0268] The server sends the photo data to an artificial intelligence model.

[0269] Once verified, the photo data is passed to the AI ​​model, which then sends the photo data and associated metadata to the AI ​​model.

[0270] Step 5:

[0271] An artificial intelligence model analyzes the two-dimensional data.

[0272] The AI ​​model identifies people and objects in the photo, analyzes their shape and depth, and uses deep learning algorithms to analyze each pixel in the photo and generate mesh data to recreate the three-dimensional structure.

[0273] Step 6:

[0274] An artificial intelligence model generates 3D meshes and textures.

[0275] A three-dimensional shape is created based on the mesh data. Color information is extracted from the original photo and texture is applied to the generated mesh. This results in a 3D model that looks realistic.

[0276] Step 7:

[0277] The server places the generated 3D data in a virtual space.

[0278] The completed 3D data is placed at a specific location in the virtual space. If multiple 3D objects are required, each object is placed in the appropriate location.

[0279] Step 8:

[0280] The server links the 3D data to the user's profile.

[0281] The generated 3D data ID is linked to the user ID and saved in a database, allowing the user to access the 3D data at any time.

[0282] Step 9:

[0283] The user puts on the virtual reality device and starts a dedicated application.

[0284] The user puts on the VR goggles and starts the Metaverse application. After the user is authenticated, a list of available albums is displayed within the application.

[0285] Step 10:

[0286] The user selects and views the generated 3D album.

[0287] Select "Family Trip" from the album list and view the 3D data placed in the virtual space. Users can change the viewpoint and walk around the virtual space.

[0288] Step 11:

[0289] A user sends a request to the server to invite others.

[0290] A user generates a request on their device to invite other users to the same virtual space and sends it to the server, which generates an invitation link or sharing code.

[0291] Step 12:

[0292] The server adds viewing permissions to other people's profiles.

[0293] The server receives the invitation request and adds the permission to view the three-dimensional data to the profile of the other specified user.

[0294] Step 13:

[0295] Other users access the virtual space using the invitation link.

[0296] Other users click on the invitation link and join the same virtual space using their virtual reality devices. The server synchronizes data in real time, and multiple avatars are displayed simultaneously.

[0297] Step 14:

[0298] The server synchronizes the location information and viewpoint data among multiple users.

[0299] The server receives each user's position and viewpoint data in real time and transmits that information to other users, allowing each user to see the avatars of other users in real time.

[0300] Step 15:

[0301] Users share experiences with other users in a virtual space.

[0302] Users can communicate with other users in a virtual space while viewing and experiencing the same 3D data.

[0303] Step 16:

[0304] The emotion recognition engine collects the user's facial expressions and voice data.

[0305] The emotion recognition engine uses the camera and microphone installed in the VR goggles to capture the user's facial expressions and voice tone, thereby grasping the user's real-time emotional state.

[0306] Step 17:

[0307] An emotion recognition engine analyzes the user's emotions.

[0308] The emotion recognition engine analyzes the collected data and determines what emotions the user is feeling, such as smiling, surprised, or sad.

[0309] Step 18:

[0310] The server reflects the recognized emotions in the virtual space.

[0311] When the user's emotion is recognized, the server applies that data to the virtual space, causing objects and effects in the virtual space to change dynamically. For example, if the user smiles, an effect that brightens the scenery is applied.

[0312] Step 19:

[0313] The server shares emotion data among multiple users.

[0314] When multiple users exist in a virtual space, the server also synchronizes each user's emotional data in real time, allowing each user to see the emotional state of the other users.

[0315] Step 20:

[0316] Other users experience the shared emotion data.

[0317] Other users in the virtual space can experience the emotional changes in real time and adjust their own actions and reactions accordingly.

[0318] This series of steps allows users to re-experience a simple two-dimensional photograph in a realistic three-dimensional space, and share the experience with other users in a virtual space that changes dynamically with changes in emotions.

[0319] Example 2

[0320] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0321] While existing technologies exist for systems that allow users to experience 2D real-world data (such as photos and images) in 3D virtual space, they lack the ability to recognize users' emotions in real time and dynamically change objects and effects in the virtual space accordingly, as well as the ability to share emotions in real time among multiple users. For this reason, there is a demand for systems that can make user experiences richer and more interactive.

[0322] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for a user to upload 2D data from a terminal, means for receiving the 2D data and verifying the data format and file size before transmitting it to an artificial intelligence model, means for analyzing the 2D data using the artificial intelligence model and converting it into 3D data, means for arranging the converted 3D data in a virtual space, means for a user to access the 3D data in the virtual space via a virtual reality device to view and experience it, means for multiple users to share and experience the 3D data in the same virtual space, means for collecting emotion data using a user's facial expression recognition camera or microphone and transmitting it to an emotion recognition engine, means for dynamically changing objects and effects in the virtual space based on the recognized emotion data, and means for sharing and reflecting emotion data with other users in real time. This enables dynamic changes in the virtual space according to the user's emotions and emotion sharing among multiple users.

[0323] A "terminal" is an electronic device used by a user, and includes forms such as a personal computer, a smartphone, and a tablet.

[0324] "Two-dimensional data" refers to data that exists on a plane, such as photographs, images, and graphics.

[0325] A "server" is a computer system that communicates with multiple terminals via a network and stores, processes, sends and receives data.

[0326] "Data format" refers to the method or structure in which digital data is stored, including image formats such as JPEG and PNG.

[0327] "File size" is an indicator of the size of data and is expressed in units such as kilobytes (KB) or megabytes (MB).

[0328] An "artificial intelligence model" refers to a software system that uses machine learning algorithms or deep learning to perform specific tasks.

[0329] "Image analysis" is the process of extracting and identifying features and patterns in two-dimensional data, and primarily uses deep learning technology.

[0330] "Three-dimensional data" is data that has depth information, including the shape and position of a three-dimensional object.

[0331] A "3D mesh" is a data structure that represents the surface of a 3D object and is composed of a large number of polygons (mainly triangles).

[0332] "Texture" refers to image or color information applied to a 3D mesh to make the object appear more realistic.

[0333] A "virtual space" is a computer-generated three-dimensional environment that users can visually experience.

[0334] "Virtual reality devices" refer to devices for experiencing virtual spaces, such as VR goggles and head-mounted displays (HMDs).

[0335] An "expression recognition camera" is a camera device that detects a user's facial expressions and captures them as data.

[0336] A "microphone" is a device for collecting sound and plays a role in transmitting the user's voice to the emotion recognition engine.

[0337] An "emotion recognition engine" is a software system that analyzes data such as facial expressions and tone of voice to recognize a user's emotional state.

[0338] "Emotional data" is data that represents the user's emotional state and is collected and analyzed in real time.

[0339] An "object" refers to any three-dimensional model or entity that exists in a virtual space.

[0340] "Effects" are visual effects that are displayed in response to specific events or actions within the virtual space.

[0341] "Location information" refers to the coordinate information of a user or avatar within a virtual space.

[0342] "Viewpoint data" is data that represents the user's line of sight and the camera direction.

[0343] This invention is a system that allows users to experience two-dimensional data in three dimensions within a virtual space and share that experience among multiple users, and also adds technology that recognizes the user's emotions and dynamically changes the objects and effects within the virtual space in response to those emotions.

[0344] An embodiment of this system includes the following process.

[0345] 1. User uploads 2D data

[0346] Users use their own devices (e.g., personal computers, smartphones, tablets) to select and upload 2D data such as photos and images from their local disks. Specifically, users click the file selection button and select the image file they want to upload. After making their selection, they press the "Upload" button, which causes the device to send a request to the server.

[0347] 2. Receipt and verification of data by the server

[0348] The server receives the upload request sent from the device. Then, it verifies the file format and size of the received 2D data. For example, if the format is invalid or the file size is too large, it returns an error message to the user.

[0349] 3. Sending data from the server to the AI ​​model

[0350] The server then sends the verified 2D data to the AI ​​model, preprocessing the data (e.g., resizing the image or converting the format). After sending, the server records the progress and may notify the user in real time.

[0351] 4. Analyzing and transforming data using artificial intelligence models

[0352] The AI ​​model uses image analysis techniques to identify the shape and depth of people and objects from 2D data. Specifically, it uses deep learning algorithms (e.g., convolutional neural networks) to extract feature points within the image. It then generates a 3D mesh based on these feature points and applies a texture based on color information extracted from the original photo.

[0353] 5. Placement of 3D data in virtual space by the server

[0354] The server places the 3D data received from the AI ​​model in the virtual space. Specifically, it places the data according to the virtual space's coordinate system and sets appropriate position information, allowing users to smoothly browse the virtual space using a virtual reality device.

[0355] 6. User Experience of Virtual Space

[0356] The user puts on the virtual reality device (e.g., VR goggles, head-mounted display) and launches a dedicated application. Specifically, they click the "Start Experience" button from the menu within the application and move into the virtual space. The user can change their viewpoint and walk around the virtual space using a controller.

[0357] 7. Shared experiences among multiple users

[0358] Users can invite other users to the virtual space by clicking the "Invite" button in the dedicated application, entering their friend's user ID, and sending an invitation link. Invited users can also use virtual reality devices and the application to experience the same virtual space. The server synchronizes the location and viewpoint data of multiple users in real time, and displays each user's avatar, enabling a shared experience.

[0359] 8. How the Emotion Recognition Engine Works

[0360] The server sends data collected from the facial recognition camera and microphone installed on the user's device to the emotion recognition engine. For example, the device uses the front camera to detect the user's facial expressions. In the case of audio data, the audio recorded by the microphone is sent. The emotion recognition engine analyzes this data and determines the user's emotional state in real time.

[0361] 9. Changes in virtual space based on emotional data

[0362] The server dynamically changes the objects and effects in the virtual space based on the emotional data obtained from the emotion recognition engine. If the user makes a surprised expression, the objects in the virtual space will move all at once, or special effects (e.g., fireworks or light effects) will be displayed.

[0363] 10. Emotion Sharing among Multiple Users

[0364] The server receives emotional data from multiple users in real time and shares it within the virtual space. User A's emotional data is also displayed to User B, allowing them to mutually confirm their emotional state. For example, if User A smiles, User B's avatar will also smile.

[0365] Reproducing specific examples

[0366] As a concrete example, we will explain the case where a photo of a family trip is used. User A uploads a group photo of the family trip, "familytrip.jpg", from his / her terminal. The detailed procedure is explained below.

[0367] Upload steps:

[0368] User A operates the terminal to select the family trip photo "familytrip.jpg" and clicks the "upload" button.

[0369] The server receives the upload request and sends it to the artificial intelligence model.

[0370] Data analysis and transformation:

[0371] An artificial intelligence model analyzes the photo, identifies the position of family members' faces and the background scenery, and generates a 3D mesh.

[0372] A texture is applied to the generated 3D mesh based on color information extracted from the original photo.

[0373] Placement in virtual space:

[0374] The server uploads the generated 3D data to the virtual space and links it to User A's profile.

[0375] Virtual space experience:

[0376] User A puts on the VR goggles, launches the Metaverse application, and selects the "Family Trip" album. In the virtual space, he or she relives memories of the family trip in a realistic way.

[0377] Emotion Recognition and Feedback:

[0378] The emotion recognition engine monitors the facial expressions and voice of user A and analyzes their emotional state in real time. For example, if user A smiles, a character in the virtual space may greet them.

[0379] Adding a shared experience:

[0380] User A invites Friend B to this virtual space, and the server grants Friend B viewing permission to join the virtual space. The two can relive memories of a family trip and share their emotions in real time. If User A makes a surprised expression, that facial expression data is transmitted to Friend B, and a surprise effect may be displayed in the virtual space.

[0381] Using this system, it is possible to realize an experience that goes beyond simple two-dimensional photographs, and instead involves a three-dimensional virtual space that changes dynamically according to the user's emotions.

[0382] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0383] Step 1: User uploads 2D data

[0384] Input: 2D data selected by the user (e.g., photo "Family Trip.jpg")

[0385] Output: Upload request from device to server

[0386] Specific operation: The user operates the device, clicks the file selection button, and selects 2D data such as a photo or image. After making a selection, the user presses the "Upload" button, which sends the selected data to the server. The device uploads this data as an HTTP request to the server in the specified format.

[0387] Step 2: Server receives and verifies data

[0388] Input: Upload request sent from the device (2D data)

[0389] Output: Validated 2D data or error message

[0390] Specific operation: After the server receives the upload request, it verifies the data format (e.g., JPEG, PNG) and file size (e.g., less than 2MB). If the format is invalid or the file size is too large, it returns an error message to the user. If the verification is successful, the received data proceeds to the next step.

[0391] Step 3: Sending data from the server to the AI ​​model

[0392] Input: Verified 2D data

[0393] Output: A request to send data to the AI ​​model

[0394] How it works: The server sends the verified 2D data to the AI ​​model. If necessary, it preprocesses the data (e.g., resizes or converts the image). The preprocessed data is sent to the AI ​​model and progress is recorded. The user may be notified of progress in real time.

[0395] Step 4: Analyze and transform the data with artificial intelligence models

[0396] Input: 2D data sent from the server

[0397] Output: 3D mesh, 3D data with texture information

[0398] How it works: The AI ​​model uses image analysis techniques to identify the shape and depth of people and objects from 2D data. It then uses deep learning algorithms (e.g., convolutional neural networks) to extract feature points within the image. This generates a 3D mesh and applies texture based on color information extracted from the original photo.

[0399] Step 5: The server places the 3D data in the virtual space.

[0400] Input: 3D data sent from an artificial intelligence model

[0401] Output: 3D data placed in virtual space

[0402] Specific operation: The server places the 3D data received from the AI ​​model in the virtual space. The server aligns the data with the virtual space's coordinate system and sets appropriate position information. This allows the user to smoothly browse the virtual space using a virtual reality device.

[0403] Step 6: User experience of the virtual space

[0404] Input: 3D data placed in virtual space

[0405] Output: User experience data (viewpoint movement, action data)

[0406] Specific operation: The user puts on the virtual reality device (e.g., VR goggles) and launches the dedicated application. They click the "Start Experience" button from the application's menu and move into the virtual space. The user walks around the virtual space, changes their viewpoint, and operates the controller. This experience data is recorded in real time.

[0407] Step 7: Shared experiences with multiple users

[0408] Input: User invitation request, location and viewpoint data of other users

[0409] Output: Synchronized experience data for multiple users

[0410] How it works: A user clicks the "Invite" button in the dedicated application, enters their friend's user ID, and sends an invitation link. The invited user also uses the virtual reality device and application to experience the same virtual space. The server synchronizes the location and viewpoint data of multiple users in real time, displays each user's avatar, and enables a shared experience.

[0411] Step 8: Emotion Recognition Engine in Action

[0412] Input: User's facial expression data and voice data

[0413] Output: Parsed emotion data

[0414] Specific operation: The server sends data collected from the facial recognition camera and microphone installed on the user's device to the emotion recognition engine. For example, the front camera is used to detect the user's facial expressions, and voice data is recorded by the microphone. The emotion recognition engine analyzes this data and determines the user's emotional state in real time.

[0415] Step 9: Change the virtual space based on emotion data

[0416] Input: Emotion data sent from the emotion recognition engine

[0417] Output: Dynamically changing virtual space objects and effects

[0418] Specific operation: The server dynamically changes the objects and effects in the virtual space based on the emotional data obtained from the emotion recognition engine. For example, if the user makes a surprised expression, the objects in the virtual space will all move in unison and special effects will be displayed.

[0419] Step 10: Emotion sharing among multiple users

[0420] Input: Emotion data from multiple users

[0421] Output: Changes in the virtual space based on shared emotional data

[0422] Specific operation: The server receives emotional data from multiple users in real time and shares it in the virtual space. For example, the emotional data of user A can be displayed to user B, allowing them to understand each other's emotional states. When user A smiles, user B's avatar also smiles. In this way, emotions can be shared among multiple users.

[0423] (Application example 2)

[0424] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0425] The present invention relates to a system that uses two-dimensional data to provide a three-dimensional experience and allows users to share that experience. However, conventional systems have the problem of being unable to dynamically change the environment or objects in the virtual space in response to the user's emotions, resulting in a lack of immersion and emotional sharing between users. Furthermore, when multiple users share an experience in a virtual reality space, the lack of real-time sharing of emotional data and changes to the environment based on those emotions limits the experience.

[0426] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for a user to upload 2D data from a terminal, means for receiving the 2D data and applying an artificial intelligence model to convert the 2D data into 3D data, means for arranging the converted 3D data in a virtual space, means for a user to experience the 3D data in the virtual space via a virtual reality device, means for multiple users to share and experience the 3D data in the same virtual space, means for recognizing a user's emotion and dynamically changing objects and effects in the virtual space based on the emotion, and means for sharing emotion data of multiple users in real time and dynamically changing effects in the virtual space based on the data. As a result, the environment and objects in the virtual space dynamically change according to the user's emotion, enabling emotions to be shared among multiple users.

[0427] "Two-dimensional data" is digital data that is represented on a plane, such as photographs, images, and videos.

[0428] "3D data" is data that has depth and a sense of three-dimensionality, and is digital data that includes meshes and textures.

[0429] An "artificial intelligence model" is an algorithm that uses machine learning and deep learning, and is a computational tool for analyzing two-dimensional data and converting it into three-dimensional data.

[0430] A "virtual space" is a computer-generated, three-dimensional digital environment that a user can experience through a virtual reality device.

[0431] A "virtual reality device" is a device that allows users to experience data in a virtual space, such as a VR headset or HMD (Head-Mounted Display).

[0432] An "emotion recognition engine" is software or hardware that analyzes a user's facial expressions and voice to determine their emotional state.

[0433] "Effects" are visual or auditory effects displayed within the virtual space, which are dynamically generated according to the user's emotional state.

[0434] A "3D mesh" is a polygon structure formed by vertices, edges, and faces that make up a three-dimensional shape, and is a data structure that provides an actual sense of three-dimensionality.

[0435] A "texture" is image data that is applied to a 3D mesh to realistically represent the texture and color of an object's surface.

[0436] An "avatar" is a digital character that represents a user in a virtual space.

[0437] "Emotion data" is data that indicates the emotional state of the user analyzed by an emotion recognition engine.

[0438] "Real-time synchronization" means that the location information, viewpoint data, emotional data, etc. of multiple users in a virtual space are updated simultaneously, allowing all users to share the latest information.

[0439] The present invention is a system that allows users to experience 2D data in 3D within a virtual space, share that experience among multiple users, and dynamically change objects and effects within the virtual space based on the user's emotions. The realization of this system involves the following means and processes.

[0440] First, users use their devices to upload two-dimensional data, which refers to digital data represented on a flat surface, such as photographs, images, and videos, which are selected and uploaded through the device's user interface.

[0441] The server then receives the uploaded 2D data, verifies the data, and then converts it into 3D data using a generative AI model. A generative AI model is an algorithm designed using machine learning or deep learning that analyzes 2D image data, estimates 3D shapes, and generates 3D meshes and textures based on the 3D shapes.

[0442] The server then places the generated 3D data in a virtual space, which is a computer-generated 3D digital environment that users can experience through a virtual reality device. Users can wear the device and view and experience the 3D data in the virtual space.

[0443] Another feature of the present invention is an emotion recognition engine that recognizes a user's emotions in real time. An emotion recognition engine is software or hardware for analyzing a user's emotional state from their facial expressions and voice. The server collects the user's facial expressions and voice and sends them to the emotion recognition engine. The emotion recognition engine analyzes this data and determines the user's emotions.

[0444] Based on the emotional data, the server dynamically changes the objects and effects in the virtual space. For example, if the user expresses surprise, objects in the virtual space may suddenly move or visual effects may be displayed. This makes the user's experience more intuitive and immersive.

[0445] Furthermore, the system includes a means for sharing emotional data among multiple users in real time. When multiple users are experiencing the same virtual space, they can share their emotional data and dynamically change the effects in the virtual space based on that data. This allows multiple users to share a common experience and feel a sense of emotional unity.

[0446] A specific use case is a scenario where a user uploads a video of a family trip, converts it into 3D, and relives it using a virtual reality device. When the emotion recognition engine recognizes the user's smile, effects such as fireworks appear in the virtual space. Friends can also be invited to the same virtual space to enjoy the experience together.

[0447] Example prompt sentence:

[0448] "Upload a video of your family trip"

[0449] "Displays various effects when the face is surprised"

[0450] "Invite friends to join you in VR"

[0451] This allows users to enjoy a richer, more shareable virtual reality experience.

[0452] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0453] Step 1:

[0454] The user uploads 2D data using the terminal. Specifically, the user selects a 2D data file such as a photo or video and clicks the upload button, and the terminal sends the selected data to the server. The input is the 2D data file, and the output is the data sent to the server.

[0455] Step 2:

[0456] The server receives the 2D data. The server verifies the format and file size of the received data to ensure it is valid. The input is the 2D data file sent from the terminal, and the output is the verified data.

[0457] Step 3:

[0458] The server sends the verified 2D data to the generative AI model, which uses machine learning algorithms to analyze the 2D data and convert it into 3D data. The input is a 2D data file, and the output is 3D data.

[0459] Step 4:

[0460] The server places the generated 3D data in a virtual space. The server then places the 3D mesh and texture in the appropriate position within the virtual space, allowing the user to experience it using a virtual reality device. The input is the 3D data output by the generative AI model, and the output is 3D data placed within the virtual space.

[0461] Step 5:

[0462] The user wears a virtual reality device and experiences 3D data in a virtual space. The user puts on a VR headset and launches a dedicated application to enter the virtual space. The input is 3D data placed in the virtual space, and the output is the user's experience.

[0463] Step 6:

[0464] The server sends the user's facial expressions and voice to the emotion recognition engine. The camera and microphone installed in the virtual reality device capture the user's facial expressions and voice in real time and send them to the server. The server then sends them to the emotion recognition engine. The input is the user's facial expression and voice data, and the output is the input data for the emotion recognition engine.

[0465] Step 7:

[0466] The emotion recognition engine determines the user's emotional state. The emotion recognition engine analyzes facial and voice data to determine the user's emotion. The input is the captured facial and voice data, and the output is the user's emotional state.

[0467] Step 8:

[0468] The server dynamically changes objects and effects in the virtual space based on the emotional data. For example, if the user makes a surprised expression, a specific effect is displayed. The input is the user's emotional data, and the output is the changed effects and objects in the virtual space.

[0469] Step 9:

[0470] Emotional data from multiple users is shared in real time. The server receives each user's emotional data and synchronizes it within the virtual space. This allows the emotions to be shared with other users, and corresponding effects are activated. The input is the emotional data of multiple users, and the output is the emotional data synchronized in real time and effects based on it.

[0471] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0472] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0473] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0474] [Second embodiment]

[0475] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0476] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0477] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0478] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0479] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0480] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0481] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0482] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0483] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0484] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0485] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0486] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0487] The present invention is a system that allows users to experience 2D data in 3D in a virtual space and share that experience among multiple users. This system mainly consists of the following steps.

[0488] 1. The user uploads 2D data

[0489] Users use their own devices to select and upload 2D data such as photos and images from their local disks, which the devices then receive and send to the server.

[0490] 2. The server receives the 2D data and sends it to the AI ​​model.

[0491] The server receives the uploaded 2D data, verifies the data format and file size, and then sends it to an AI model, which uses machine learning algorithms to analyze the 2D data and convert it into 3D data.

[0492] 3. Artificial intelligence models transform 2D data into 3D

[0493] The AI ​​model uses image analysis technology to identify the shape and depth of people and objects from 2D data. Based on this information, it generates a 3D mesh and applies textures to it, resulting in realistic 3D data.

[0494] 4. The server places the 3D data in the virtual space

[0495] The generated 3D data is placed in a virtual space by the server, and the space is ready for the user to view using a virtual reality device.

[0496] 5. The user experiences the virtual space using a virtual reality device.

[0497] Users wear virtual reality devices and launch a dedicated application to view and experience 3D data placed in a virtual space. Users can change their viewpoint and walk around the virtual space.

[0498] 6. Shared experiences among multiple users

[0499] Users can invite other users to the virtual space. Invited users can also use the same virtual reality device and application to experience the same virtual space. The server synchronizes the location and viewpoint data of multiple users in real time and displays each user's avatar, enabling a shared experience.

[0500] Specific examples

[0501] As a specific example, a case where photos of a family trip are used will be described.

[0502] User A: Uploads a group photo of a family trip from the device. The photo is a file called "familytrip.jpg." User A operates the device, selects the photo, and clicks the "Upload" button.

[0503] Server: Receives the upload request, saves "Family Trip.jpg", and sends it to the AI ​​model, which analyzes the photo, identifies the positions of family faces and the background scenery, and generates a 3D mesh.

[0504] Artificial intelligence model: Generates a 3D mesh and applies texture based on color information extracted from the original photo.

[0505] Server: Uploads the generated 3D data to the virtual space and links it to User A's profile.

[0506] User A: Puts on the VR goggles, launches the Metaverse application, and selects the "Family Trip" album. User A relives the family trip in a virtual space in a realistic way.

[0507] User A: Furthermore, User B invites Friend B to this virtual space. The server grants Friend B permission to view the space, and Friend B also puts on VR goggles and joins the virtual space. The two can relive memories of their family trip together and communicate in real time.

[0508] This system allows users to not only memorize memorable scenes as photographs, but also recreate them in three dimensions, enabling shared experiences in real time.

[0509] The processing flow will be explained below.

[0510] Step 1:

[0511] The user uses the terminal to select two-dimensional data (e.g., a photo file).

[0512] The user opens a browser or a dedicated application, accesses the photo upload screen, selects a file such as "Family Trip.jpg" from the local disk, and clicks the upload button.

[0513] Step 2:

[0514] The device sends the selected two-dimensional data to the server.

[0515] The device generates an HTTP POST request and sends the selected photo data and user ID to the server, along with metadata such as file name, file format, and file size.

[0516] Step 3:

[0517] The server receives and verifies the 2D data.

[0518] The server receives the HTTP POST request, saves the photo data in temporary storage, and performs validation checks such as checking for valid file formats and file sizes, and virus checks.

[0519] Step 4:

[0520] The server sends the photo data to an artificial intelligence model.

[0521] Once verified, the photo data is passed to the AI ​​model, which then sends the photo data and associated metadata to the AI ​​model.

[0522] Step 5:

[0523] An artificial intelligence model analyzes the two-dimensional data.

[0524] The AI ​​model identifies people and objects in the photo, analyzes their shape and depth, and uses deep learning algorithms to analyze each pixel in the photo and generate mesh data to recreate the three-dimensional structure.

[0525] Step 6:

[0526] An artificial intelligence model generates 3D meshes and textures.

[0527] A three-dimensional shape is created based on the mesh data. Color information is extracted from the original photo and texture is applied to the generated mesh. This results in a 3D model that looks realistic.

[0528] Step 7:

[0529] The server places the generated 3D data in a virtual space.

[0530] The completed 3D data is placed at a specific location in the virtual space. If multiple 3D objects are required, each object is placed in the appropriate location.

[0531] Step 8:

[0532] The server links the 3D data to the user's profile.

[0533] The generated 3D data ID is linked to the user ID and saved in a database, allowing the user to access the 3D data at any time.

[0534] Step 9:

[0535] The user puts on the virtual reality device and starts a dedicated application.

[0536] The user puts on the VR goggles and starts the Metaverse application. After the user is authenticated, a list of available albums is displayed within the application.

[0537] Step 10:

[0538] The user selects and views the generated 3D album.

[0539] Select "Family Trip" from the album list and view the 3D data placed in the virtual space. Users can change the viewpoint and walk around the virtual space.

[0540] Step 11:

[0541] A user sends a request to the server to invite others.

[0542] A user generates a request on their device to invite other users to the same virtual space and sends it to the server, which generates an invitation link or sharing code.

[0543] Step 12:

[0544] The server adds viewing permissions to other people's profiles.

[0545] The server receives the invitation request and adds the permission to view the three-dimensional data to the profile of the other specified user.

[0546] Step 13:

[0547] Other users access the virtual space using the invitation link.

[0548] Other users click on the invitation link and join the same virtual space using their virtual reality devices. The server synchronizes data in real time, and multiple avatars are displayed simultaneously.

[0549] Step 14:

[0550] The server synchronizes the location information and viewpoint data among multiple users.

[0551] The server receives each user's position and viewpoint data in real time and transmits that information to other users, allowing each user to see the avatars of other users in real time.

[0552] This series of steps allows users to re-experience a simple two-dimensional photo in a realistic three-dimensional space and share it with other users.

[0553] Example 1

[0554] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0555] Modern virtual reality technology is often experienced by users alone, and individual experiences tend to be fragmented. Furthermore, the process of converting 2D data into 3D and placing it in a virtual space is extremely complex and requires extensive specialized knowledge. As a result, it is difficult for ordinary users to create realistic 3D experiences from 2D data and share them with other users. Furthermore, it is difficult for multiple users to share an experience in real time in the same virtual space, limiting collaborative experiences.

[0556] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0557] In this invention, the server includes a means for users to upload 2D data from their devices, a means for receiving the 2D data and applying a generative AI model to convert it into 3D data, and a means for placing the converted 3D data in a virtual space. This allows general users to easily convert 2D data into 3D data, create realistic virtual experiences, and share them with other users. It also allows multiple users to experience 3D data in real time within the same virtual space, synchronizing each user's location information and viewpoint data. As a result, users can enjoy a collaborative virtual experience, enabling virtual reality technology to be enjoyed on a wider scale.

[0558] A "user" is a person who uses a terminal to upload two-dimensional data and experience virtual reality.

[0559] A "terminal" refers to an electronic device used by a user, such as a computer or smartphone.

[0560] "Two-dimensional data" refers to two-dimensional information such as photos and image data uploaded by users.

[0561] "Upload" refers to the act of a user sending data from a terminal to a server.

[0562] A "server" is a computer system that processes two-dimensional data received from users, converts it into three-dimensional data, and places it in a virtual space.

[0563] A "generative AI model" is a program that uses artificial intelligence technology to analyze two-dimensional data and convert it into three-dimensional data.

[0564] "3D data" refers to three-dimensional data converted from 2D data by a generative AI model.

[0565] "Virtual space" refers to a three-dimensional virtual space generated by computer graphics.

[0566] A "virtual reality device" refers to a head-mounted display or VR goggles that users use to experience a virtual space.

[0567] A "shared experience" is when multiple users simultaneously experience virtual reality in the same virtual space.

[0568] "Location information" refers to coordinate data within the user's virtual space.

[0569] "Viewpoint data" refers to data that indicates the visual direction and angle of the user within the virtual space.

[0570] An "invitation request" is a request made by a user to invite another user to a virtual space.

[0571] "Viewing authority" refers to the authority that allows an invited user to view data in a virtual space.

[0572] An "avatar" is a three-dimensional character that represents a user in a virtual space.

[0573] "Real-time synchronization" refers to the simultaneous updating of location information and viewpoint data for multiple users within a virtual space.

[0574] The present invention is a system that allows users to upload 2D data from their devices, convert it into 3D data, and share it in a virtual space. The steps for implementing this system are as follows:

[0575] First, the user uploads 2D data using a terminal. The terminal can be an electronic device such as a PC or smartphone. The user opens a file selection dialog, selects a photo or image data, such as "Family Trip.jpg," from the local disk, and clicks the upload button to send the data to the server. The technology used is an HTTP POST request.

[0576] Next, the server receives the 2D data. Using a web framework such as Flask or Django, the server stores the data appropriately and validates the data format and file size. The server then sends the 2D data to the artificial intelligence model for generative AI model application. The server uses a REST API or JSON format for this purpose.

[0577] The generative AI model uses machine learning libraries such as TensorFlow and PyTorch to analyze 2D data and convert it into 3D data. The AI ​​model analyzes the received 2D data to extract object contours and estimate depth. Based on the results of this analysis, it generates a 3D mesh and applies texture using color information extracted from the original image.

[0578] The server receives the generated 3D data and places it in a virtual space. The virtual space is built using a virtual space engine such as Unity or Unreal Engine. The server stores the generated 3D data in a database and places it appropriately in the virtual space.

[0579] Users wear virtual reality devices (such as Oculus Rift or HTC Vive) and view and experience 3D data in a virtual space. By launching a dedicated application, users can move freely within the virtual space and interact with it.

[0580] In addition, multiple users can share the experience of 3D data in the same virtual space. Users can invite other users to the virtual space, and the invited users also use virtual reality devices. The server synchronizes the location and viewpoint data of multiple users in real time and displays the avatars of other users.

[0581] Specific examples

[0582] As a concrete example, let's consider the case of a family trip photo. User A uploads a group photo of the family trip, "familytrip.jpg," from their device. The server receives the upload request, saves "familytrip.jpg," and sends it to the generative AI model. The generative AI model analyzes the photo, identifies the position of family members' faces and the background scenery, and generates a 3D mesh. The server uploads the generated 3D data to the virtual space and links it to User A's profile. User A puts on VR goggles, launches the Metaverse application, and selects the "Family Trip" album. Within the virtual space, they relive their family trip memories in a way that is close to reality. User A invites Friend B to this virtual space, and the server grants Friend B viewing permissions as well. Friend B also puts on VR goggles and joins the virtual space, and the two can experience their family trip memories together and communicate in real time.

[0583] Prompt Sentence Examples

[0584] What are the steps to upload photos from a family trip, turn them into 3D, experience them in a virtual space using a VR device, and share them with friends?

[0585] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0586] Step 1:

[0587] The user selects an image on the device

[0588] The user opens a file selection dialog using the terminal and selects an image to upload from the local disk (for example, "Family Trip.jpg"). The input is the 2D data (image file) selected by the user. The output is the path information of the selected image file.

[0589] Step 2:

[0590] The device sends the data to the server

[0591] When the user clicks the "Upload" button, the device sends the selected image data to the server as an HTTP POST request. Specifically, it uses the JavaScript fetch API to send the image data as form data. The input is the path information of the selected image file, and the output is the image data sent to the server.

[0592] Step 3:

[0593] The server receives the data

[0594] The server receives the image data sent from the device. It processes the HTTP POST request using a web framework such as Flask or Django. Specifically, the server temporarily stores the image data and verifies the format and size. The input is the image data in the HTTP request, and the output is the verified image data.

[0595] Step 4:

[0596] The server sends the data to the artificial intelligence model

[0597] The server sends the received image data to the generative AI model. Specifically, it uses a REST API to send the image data in JSON format. The input is the verified image data, and the output is an analysis request to the artificial intelligence model.

[0598] Step 5:

[0599] An artificial intelligence model analyzes the data

[0600] The generative AI model uses TensorFlow and PyTorch to perform image analysis. Specifically, it performs object recognition and depth estimation in the image to generate a 3D mesh. The input is 2D image data sent from the server, and the output is 3D mesh data.

[0601] Step 6:

[0602] Applying textures to 3D meshes

[0603] The AI ​​model applies texture to the generated 3D mesh using color information extracted from the original image. Specifically, it uses texture mapping technology. The input is 3D mesh data and texture information, and the output is the completed 3D data.

[0604] Step 7:

[0605] The server places the 3D data in the virtual space

[0606] The server receives the generated 3D data and places it in a virtual space created with Unity or Unreal Engine. Specifically, it stores the data in a database and transfers it to the virtual space engine. The input is the completed 3D data, and the output is 3D data placed in the virtual space.

[0607] Step 8:

[0608] Users experience a virtual space using a virtual reality device.

[0609] The user puts on the VR goggles and launches a dedicated application. Specific actions include visually viewing and manipulating objects in a virtual space. The input is 3D data placed in the virtual space, and the output is the visual information and interaction information of the virtual reality that the user experiences.

[0610] Step 9:

[0611] Shared experiences among multiple users

[0612] Users can invite other users into the virtual space, and the server synchronizes each user's location information and viewpoint data in real time. Specifically, the invited users also wear the same virtual reality device and use real-time communication technology. The input is the location information and viewpoint data of multiple users, and the output is a real-time shared virtual space experience.

[0613] Prompt Sentence Examples

[0614] What are the steps to upload photos from a family trip, turn them into 3D, experience them in a virtual space using a VR device, and share them with friends?

[0615] (Application example 1)

[0616] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0617] The present invention relates to a system that not only allows users to experience two-dimensional data in three dimensions within a virtual space, but also allows multiple users to share that experience in real time. In particular, the objective of the present invention is to provide a system that allows users to enjoy product experiences in a virtual store in three dimensions and share shopping experiences with other users.

[0618] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0619] In this invention, the server includes means for users to upload 2D data from their terminals, means for receiving the 2D data and applying an artificial intelligence model to convert it into 3D data, means for arranging the converted 3D data in a virtual space, means for users to experience the 3D data in the virtual space through a virtual reality device, means for multiple users to share the experience of the 3D data in the same virtual space, means for users to upload product images and view and experience products in a 3D manner in a virtual store, and means for inviting other users to the virtual store and sharing the shopping experience in real time, thereby enabling users to experience products in a 3D manner in the virtual store and share that experience with other users.

[0620] A "terminal" refers to an electronic device used by a user, such as a computer, smartphone, or tablet.

[0621] "Two-dimensional data" refers to digital data that contains two-dimensional information in terms of length and width, such as images and photographs.

[0622] "Upload" refers to the action of sending data from a user's terminal to a server.

[0623] "3D data" refers to three-dimensional digital data that has depth.

[0624] An "artificial intelligence model" is a data analysis technology that uses machine learning algorithms and has the ability to convert two-dimensional data into three-dimensional data.

[0625] A "virtual space" is a virtual environment that differs from reality and is constructed on a computer using digital technology.

[0626] A "virtual reality device" is a device that allows users to have an experience similar to the real world in a digital space, such as VR goggles or a head-mounted display.

[0627] "Product images" refer to photographs or digital images of products used in online shopping, etc.

[0628] A "virtual store" is a virtual shop that exists on the Internet where you can browse and purchase products.

[0629] "Viewing in 3D" refers to the act of using 3D data to display depth information, providing an experience that feels as if you are looking at the real thing.

[0630] A "shopping experience" is a series of online activities in which a user browses products and considers a purchase.

[0631] A "shared experience" refers to multiple users experiencing the same experience in real time in the same virtual space.

[0632] "Real-time" refers to processing and information exchange occurring immediately, without delay.

[0633] System Overview

[0634] The present invention relates to a system that allows users to experience two-dimensional data in a three-dimensional virtual space and share that experience among multiple users. In particular, the present invention provides a system that allows users to view products in a virtual store in three dimensions and share their shopping experience with other users.

[0635] Hardware and Software Used

[0636] Server: Use a high performance server, e.g. AWS EC2.

[0637] Virtual reality devices: Users use VR goggles or head-mounted displays. Example: Oculus Rift.

[0638] Devices: Smartphones, smart glasses, head-mounted displays, etc. Examples: Google Glass, HoloLens.

[0639] Machine learning libraries: TensorFlow, PyTorch.

[0640] 3D modeling software: Blender, Unity engine.

[0641] Details of data processing and calculation

[0642] 1. Data upload

[0643] The user uploads a product image using their device. For example, they send "New Product Bag.jpg" from their smartphone to the server. The API used for this is the REST API.

[0644] 2. Image analysis and 3D conversion

[0645] The server analyzes the received product images using machine learning algorithms. It uses a convolutional neural network (CNN)-based model to identify the shape and depth of the object in the image, then generates a 3D mesh and applies textures to it.

[0646] 3. Placement in virtual space

[0647] The server places the generated 3D data in the virtual store using the Unity engine, which optimally positions the 3D model in the virtual space.

[0648] 4. Virtual Experiences

[0649] Users wear VR goggles and experience a virtual store through the application. The server synchronizes the user's viewpoint data and location information in real time, allowing users to freely walk around the virtual store and experience the products in three dimensions.

[0650] 5. Shared Experiences

[0651] Users can invite other users to the virtual store. The server receives the invitation request and grants viewing privileges to other users. Location and viewpoint data are synchronized in real time between multiple users, allowing them to share their shopping experience.

[0652] Specific examples

[0653] User A uploads "New Product Bag.jpg" from their smartphone.

[0654] After receiving the image, the server converts it into a 3D bag using an artificial intelligence model and places it inside the virtual store.

[0655] User A puts on VR goggles and experiences the sensation of actually picking up a bag in a virtual store.

[0656] User A invites friend B to a virtual store and they shop together while viewing new products in real time.

[0657] Prompt Sentence Examples

[0658] Design an AI model that generates a 3D model from an image. The input is a product image file (e.g., "New Product Bag.jpg"), and the output is a 3D mesh and texture for display in a virtual space. The model must identify shape and depth from the image and generate a realistic 3D model. The 3D model must also be optimized for effective display in a virtual store.

[0659] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0660] Step 1:

[0661] The user uploads 2D data from their device. The input is a product image (e.g., "New Product Bag.jpg"), and the output is the image sent to the server. This data is sent to the server and received using a REST API.

[0662] Step 2:

[0663] The server validates the received 2D data. The input is the uploaded image file, and the output is the validation result to see if the data format and file size are correct. If the data is not in the correct format at this step, an error message is returned.

[0664] Step 3:

[0665] The server sends the data to an artificial intelligence model, which converts it into 3D data. The input is verified 2D data, and the output is 3D data with 3D mesh and texture. The model used here is a Convolutional Neural Network (CNN)-based analysis algorithm that identifies shape and depth to generate a 3D model.

[0666] Step 4:

[0667] The server places the generated 3D data in a virtual space. The input is 3D data, and the output is 3D data placed at specific coordinates in the virtual space. The Unity engine is used to render the 3D model in the virtual space.

[0668] Step 5:

[0669] Users wear virtual reality devices and experience 3D data in a virtual space. The input is the user's viewpoint data and position information, and the output is an interactive experience in the virtual space. The server synchronizes this data in real time, allowing users to move freely within the virtual space.

[0670] Step 6:

[0671] Users can invite other users to a virtual space to share experiences. The input is an invitation request and the other user's ID, and the output is the granting of viewing permissions to the other users. Upon receiving the invitation, the server synchronizes the location and viewpoint data of multiple users in real time, providing a shared experience in the virtual space.

[0672] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0673] This invention is a system that allows users to experience 2D data in 3D in a virtual space and share that experience with multiple users, and also adds technology that recognizes the user's emotions and dynamically changes the objects and effects in the virtual space according to the emotions. This system mainly consists of the following steps.

[0674] 1. The user uploads 2D data

[0675] Users use their own devices to select and upload 2D data such as photos and images from their local disks, which the devices then receive and send to the server.

[0676] 2. The server receives the 2D data and sends it to the AI ​​model.

[0677] The server receives the uploaded 2D data, verifies the data format and file size, and then sends it to an AI model, which uses machine learning algorithms to analyze the 2D data and convert it into 3D data.

[0678] 3. Artificial intelligence models transform 2D data into 3D

[0679] The AI ​​model uses image analysis technology to identify the shape and depth of people and objects from 2D data. Based on this information, it generates a 3D mesh and applies textures to it, resulting in realistic 3D data.

[0680] 4. The server places the 3D data in the virtual space

[0681] The generated 3D data is placed in a virtual space by the server, and the space is ready for the user to view using a virtual reality device.

[0682] 5. The user experiences the virtual space using a virtual reality device.

[0683] Users wear virtual reality devices and launch a dedicated application to view and experience 3D data placed in a virtual space. Users can change their viewpoint and walk around the virtual space.

[0684] 6. Shared experiences among multiple users

[0685] Users can invite other users to the virtual space. Invited users can also use the same virtual reality device and application to experience the same virtual space. The server synchronizes the location and viewpoint data of multiple users in real time and displays each user's avatar, enabling a shared experience.

[0686] 7. Introducing an emotion recognition engine

[0687] The user's facial expression is captured using a camera and microphone, and the captured facial expressions and vocal tone are sent to an emotion recognition engine, which analyzes the data and determines the user's emotional state in real time.

[0688] 8. Changes in virtual space based on emotional data

[0689] Based on the recognized emotion data, objects and effects in the virtual space are dynamically changed. For example, if the user makes a surprised expression, all the objects in the virtual space will move in unison, or special effects will be displayed.

[0690] 9. Emotion Sharing among Multiple Users

[0691] When multiple users are in the same virtual space, they share their emotional data in real time. The server receives this emotional data and reflects it on each user's avatar and environment, making it easier for other users to see their emotional state.

[0692] Specific examples

[0693] As a specific example, a case where photos of a family trip are used will be described.

[0694] User A: Uploads a group photo of a family trip from the device. The photo is a file called "familytrip.jpg." User A operates the device, selects the photo, and clicks the "Upload" button.

[0695] Server: Receives the upload request, saves "Family Trip.jpg", and sends it to the AI ​​model, which analyzes the photo, identifies the positions of family faces and the background scenery, and generates a 3D mesh.

[0696] Artificial intelligence model: Generates a 3D mesh and applies texture based on color information extracted from the original photo.

[0697] Server: Uploads the generated 3D data to the virtual space and links it to User A's profile.

[0698] User A: Puts on the VR goggles, launches the Metaverse application, and selects the "Family Trip" album. User A relives the family trip in a virtual space in a realistic way.

[0699] Emotion recognition engine: Monitors user A's facial expressions and voice and analyzes their emotional state in real time. For example, if user A smiles, a character in the virtual space may greet them.

[0700] User A: Then, he invites Friend B to join the virtual space. The server grants Friend B permission to view the content, and Friend B also puts on VR goggles and joins the virtual space. The two can relive memories of their family trip together and share their emotions in real time. For example, if User A makes a surprised expression, that facial expression data is transmitted to Friend B, and a surprise effect may be displayed in the virtual space.

[0701] This system allows users to not only memorize memorable scenes as two-dimensional photographs, but also recreate them in three dimensions, allowing them to be experienced and shared in a virtual space that changes dynamically according to the user's emotions.

[0702] The processing flow will be explained below.

[0703] Step 1:

[0704] The user uses the terminal to select two-dimensional data (e.g., a photo file).

[0705] The user opens a browser or a dedicated application, accesses the photo upload screen, selects a file such as "Family Trip.jpg" from the local disk, and clicks the upload button.

[0706] Step 2:

[0707] The device sends the selected two-dimensional data to the server.

[0708] The device generates an HTTP POST request and sends the selected photo data and user ID to the server, along with metadata such as file name, file format, and file size.

[0709] Step 3:

[0710] The server receives and verifies the 2D data.

[0711] The server receives the HTTP POST request, saves the photo data in temporary storage, and performs validation checks such as checking for valid file formats and file sizes, and virus checks.

[0712] Step 4:

[0713] The server sends the photo data to an artificial intelligence model.

[0714] Once verified, the photo data is passed to the AI ​​model, which then sends the photo data and associated metadata to the AI ​​model.

[0715] Step 5:

[0716] An artificial intelligence model analyzes the two-dimensional data.

[0717] The AI ​​model identifies people and objects in the photo, analyzes their shape and depth, and uses deep learning algorithms to analyze each pixel in the photo and generate mesh data to recreate the three-dimensional structure.

[0718] Step 6:

[0719] An artificial intelligence model generates 3D meshes and textures.

[0720] A three-dimensional shape is created based on the mesh data. Color information is extracted from the original photo and texture is applied to the generated mesh. This results in a 3D model that looks realistic.

[0721] Step 7:

[0722] The server places the generated 3D data in a virtual space.

[0723] The completed 3D data is placed at a specific location in the virtual space. If multiple 3D objects are required, each object is placed in the appropriate location.

[0724] Step 8:

[0725] The server links the 3D data to the user's profile.

[0726] The generated 3D data ID is linked to the user ID and saved in a database, allowing the user to access the 3D data at any time.

[0727] Step 9:

[0728] The user puts on the virtual reality device and starts a dedicated application.

[0729] The user puts on the VR goggles and starts the Metaverse application. After the user is authenticated, a list of available albums is displayed within the application.

[0730] Step 10:

[0731] The user selects and views the generated 3D album.

[0732] Select "Family Trip" from the album list and view the 3D data placed in the virtual space. Users can change the viewpoint and walk around the virtual space.

[0733] Step 11:

[0734] A user sends a request to the server to invite others.

[0735] A user generates a request on their device to invite other users to the same virtual space and sends it to the server, which generates an invitation link or sharing code.

[0736] Step 12:

[0737] The server adds viewing permissions to other people's profiles.

[0738] The server receives the invitation request and adds the permission to view the three-dimensional data to the profile of the other specified user.

[0739] Step 13:

[0740] Other users access the virtual space using the invitation link.

[0741] Other users click on the invitation link and join the same virtual space using their virtual reality devices. The server synchronizes data in real time, and multiple avatars are displayed simultaneously.

[0742] Step 14:

[0743] The server synchronizes the location information and viewpoint data among multiple users.

[0744] The server receives each user's position and viewpoint data in real time and transmits that information to other users, allowing each user to see the avatars of other users in real time.

[0745] Step 15:

[0746] Users share experiences with other users in a virtual space.

[0747] Users can communicate with other users in a virtual space while viewing and experiencing the same 3D data.

[0748] Step 16:

[0749] The emotion recognition engine collects the user's facial expressions and voice data.

[0750] The emotion recognition engine uses the camera and microphone installed in the VR goggles to capture the user's facial expressions and voice tone, thereby grasping the user's real-time emotional state.

[0751] Step 17:

[0752] An emotion recognition engine analyzes the user's emotions.

[0753] The emotion recognition engine analyzes the collected data and determines what emotions the user is feeling, such as smiling, surprised, or sad.

[0754] Step 18:

[0755] The server reflects the recognized emotions in the virtual space.

[0756] When the user's emotion is recognized, the server applies that data to the virtual space, causing objects and effects in the virtual space to change dynamically. For example, if the user smiles, an effect that brightens the scenery is applied.

[0757] Step 19:

[0758] The server shares emotion data among multiple users.

[0759] When multiple users exist in a virtual space, the server also synchronizes each user's emotional data in real time, allowing each user to see the emotional state of the other users.

[0760] Step 20:

[0761] Other users experience the shared emotion data.

[0762] Other users in the virtual space can experience the emotional changes in real time and adjust their own actions and reactions accordingly.

[0763] This series of steps allows users to re-experience a simple two-dimensional photograph in a realistic three-dimensional space, and share the experience with other users in a virtual space that changes dynamically with changes in emotions.

[0764] Example 2

[0765] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0766] While existing technologies exist for systems that allow users to experience 2D real-world data (such as photos and images) in 3D virtual space, they lack the ability to recognize users' emotions in real time and dynamically change objects and effects in the virtual space accordingly, as well as the ability to share emotions in real time among multiple users. For this reason, there is a demand for systems that can make user experiences richer and more interactive.

[0767] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for a user to upload 2D data from a terminal, means for receiving the 2D data and verifying the data format and file size before transmitting it to an artificial intelligence model, means for analyzing the 2D data using the artificial intelligence model and converting it into 3D data, means for arranging the converted 3D data in a virtual space, means for a user to access the 3D data in the virtual space via a virtual reality device to view and experience it, means for multiple users to share and experience the 3D data in the same virtual space, means for collecting emotion data using a user's facial expression recognition camera or microphone and transmitting it to an emotion recognition engine, means for dynamically changing objects and effects in the virtual space based on the recognized emotion data, and means for sharing and reflecting emotion data with other users in real time. This enables dynamic changes in the virtual space according to the user's emotions and emotion sharing among multiple users.

[0768] A "terminal" is an electronic device used by a user, and includes forms such as a personal computer, a smartphone, and a tablet.

[0769] "Two-dimensional data" refers to data that exists on a plane, such as photographs, images, and graphics.

[0770] A "server" is a computer system that communicates with multiple terminals via a network and stores, processes, sends and receives data.

[0771] "Data format" refers to the method or structure in which digital data is stored, including image formats such as JPEG and PNG.

[0772] "File size" is an indicator of the size of data and is expressed in units such as kilobytes (KB) or megabytes (MB).

[0773] An "artificial intelligence model" refers to a software system that uses machine learning algorithms or deep learning to perform specific tasks.

[0774] "Image analysis" is the process of extracting and identifying features and patterns in two-dimensional data, and primarily uses deep learning technology.

[0775] "Three-dimensional data" is data that has depth information, including the shape and position of a three-dimensional object.

[0776] A "3D mesh" is a data structure that represents the surface of a 3D object and is composed of a large number of polygons (mainly triangles).

[0777] "Texture" refers to image or color information applied to a 3D mesh to make the object appear more realistic.

[0778] A "virtual space" is a computer-generated three-dimensional environment that users can visually experience.

[0779] "Virtual reality devices" refer to devices for experiencing virtual spaces, such as VR goggles and head-mounted displays (HMDs).

[0780] An "expression recognition camera" is a camera device that detects a user's facial expressions and captures them as data.

[0781] A "microphone" is a device for collecting sound and plays a role in transmitting the user's voice to the emotion recognition engine.

[0782] An "emotion recognition engine" is a software system that analyzes data such as facial expressions and tone of voice to recognize a user's emotional state.

[0783] "Emotional data" is data that represents the user's emotional state and is collected and analyzed in real time.

[0784] An "object" refers to any three-dimensional model or entity that exists in a virtual space.

[0785] "Effects" are visual effects that are displayed in response to specific events or actions within the virtual space.

[0786] "Location information" refers to the coordinate information of a user or avatar within a virtual space.

[0787] "Viewpoint data" is data that represents the user's line of sight and the camera direction.

[0788] This invention is a system that allows users to experience two-dimensional data in three dimensions within a virtual space and share that experience among multiple users, and also adds technology that recognizes the user's emotions and dynamically changes the objects and effects within the virtual space in response to those emotions.

[0789] An embodiment of this system includes the following process.

[0790] 1. User uploads 2D data

[0791] Users use their own devices (e.g., personal computers, smartphones, tablets) to select and upload 2D data such as photos and images from their local disks. Specifically, users click the file selection button and select the image file they want to upload. After making their selection, they press the "Upload" button, which causes the device to send a request to the server.

[0792] 2. Receipt and verification of data by the server

[0793] The server receives the upload request sent from the device. Then, it verifies the file format and size of the received 2D data. For example, if the format is invalid or the file size is too large, it returns an error message to the user.

[0794] 3. Sending data from the server to the AI ​​model

[0795] The server then sends the verified 2D data to the AI ​​model, preprocessing the data (e.g., resizing the image or converting the format). After sending, the server records the progress and may notify the user in real time.

[0796] 4. Analyzing and transforming data using artificial intelligence models

[0797] The AI ​​model uses image analysis techniques to identify the shape and depth of people and objects from 2D data. Specifically, it uses deep learning algorithms (e.g., convolutional neural networks) to extract feature points within the image. It then generates a 3D mesh based on these feature points and applies a texture based on color information extracted from the original photo.

[0798] 5. Placement of 3D data in virtual space by the server

[0799] The server places the 3D data received from the AI ​​model in the virtual space. Specifically, it places the data according to the virtual space's coordinate system and sets appropriate position information, allowing users to smoothly browse the virtual space using a virtual reality device.

[0800] 6. User Experience of Virtual Space

[0801] The user puts on the virtual reality device (e.g., VR goggles, head-mounted display) and launches a dedicated application. Specifically, they click the "Start Experience" button from the menu within the application and move into the virtual space. The user can change their viewpoint and walk around the virtual space using a controller.

[0802] 7. Shared experiences among multiple users

[0803] Users can invite other users to the virtual space by clicking the "Invite" button in the dedicated application, entering their friend's user ID, and sending an invitation link. Invited users can also use virtual reality devices and the application to experience the same virtual space. The server synchronizes the location and viewpoint data of multiple users in real time, and displays each user's avatar, enabling a shared experience.

[0804] 8. How the Emotion Recognition Engine Works

[0805] The server sends data collected from the facial recognition camera and microphone installed on the user's device to the emotion recognition engine. For example, the device uses the front camera to detect the user's facial expressions. In the case of audio data, the audio recorded by the microphone is sent. The emotion recognition engine analyzes this data and determines the user's emotional state in real time.

[0806] 9. Changes in virtual space based on emotional data

[0807] The server dynamically changes the objects and effects in the virtual space based on the emotional data obtained from the emotion recognition engine. If the user makes a surprised expression, the objects in the virtual space will move all at once, or special effects (e.g., fireworks or light effects) will be displayed.

[0808] 10. Emotion Sharing among Multiple Users

[0809] The server receives emotional data from multiple users in real time and shares it within the virtual space. User A's emotional data is also displayed to User B, allowing them to mutually confirm their emotional state. For example, if User A smiles, User B's avatar will also smile.

[0810] Reproducing specific examples

[0811] As a concrete example, we will explain the case where a photo of a family trip is used. User A uploads a group photo of the family trip, "familytrip.jpg", from his / her terminal. The detailed procedure is explained below.

[0812] Upload steps:

[0813] User A operates the terminal to select the family trip photo "familytrip.jpg" and clicks the "upload" button.

[0814] The server receives the upload request and sends it to the artificial intelligence model.

[0815] Data analysis and transformation:

[0816] An artificial intelligence model analyzes the photo, identifies the position of family members' faces and the background scenery, and generates a 3D mesh.

[0817] A texture is applied to the generated 3D mesh based on color information extracted from the original photo.

[0818] Placement in virtual space:

[0819] The server uploads the generated 3D data to the virtual space and links it to User A's profile.

[0820] Virtual space experience:

[0821] User A puts on the VR goggles, launches the Metaverse application, and selects the "Family Trip" album. In the virtual space, he or she relives memories of the family trip in a realistic way.

[0822] Emotion Recognition and Feedback:

[0823] The emotion recognition engine monitors the facial expressions and voice of user A and analyzes their emotional state in real time. For example, if user A smiles, a character in the virtual space may greet them.

[0824] Adding a shared experience:

[0825] User A invites Friend B to this virtual space, and the server grants Friend B viewing permission to join the virtual space. The two can relive memories of a family trip and share their emotions in real time. If User A makes a surprised expression, that facial expression data is transmitted to Friend B, and a surprise effect may be displayed in the virtual space.

[0826] Using this system, it is possible to realize an experience that goes beyond simple two-dimensional photographs, and instead involves a three-dimensional virtual space that changes dynamically according to the user's emotions.

[0827] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0828] Step 1: User uploads 2D data

[0829] Input: 2D data selected by the user (e.g., photo "Family Trip.jpg")

[0830] Output: Upload request from device to server

[0831] Specific operation: The user operates the device, clicks the file selection button, and selects 2D data such as a photo or image. After making a selection, the user presses the "Upload" button, which sends the selected data to the server. The device uploads this data as an HTTP request to the server in the specified format.

[0832] Step 2: Server receives and verifies data

[0833] Input: Upload request sent from the device (2D data)

[0834] Output: Validated 2D data or error message

[0835] Specific operation: After the server receives the upload request, it verifies the data format (e.g., JPEG, PNG) and file size (e.g., less than 2MB). If the format is invalid or the file size is too large, it returns an error message to the user. If the verification is successful, the received data proceeds to the next step.

[0836] Step 3: Sending data from the server to the AI ​​model

[0837] Input: Verified 2D data

[0838] Output: A request to send data to the AI ​​model

[0839] How it works: The server sends the verified 2D data to the AI ​​model. If necessary, it preprocesses the data (e.g., resizes or converts the image). The preprocessed data is sent to the AI ​​model and progress is recorded. The user may be notified of progress in real time.

[0840] Step 4: Analyze and transform the data with artificial intelligence models

[0841] Input: 2D data sent from the server

[0842] Output: 3D mesh, 3D data with texture information

[0843] How it works: The AI ​​model uses image analysis techniques to identify the shape and depth of people and objects from 2D data. It then uses deep learning algorithms (e.g., convolutional neural networks) to extract feature points within the image. This generates a 3D mesh and applies texture based on color information extracted from the original photo.

[0844] Step 5: The server places the 3D data in the virtual space.

[0845] Input: 3D data sent from an artificial intelligence model

[0846] Output: 3D data placed in virtual space

[0847] Specific operation: The server places the 3D data received from the AI ​​model in the virtual space. The server aligns the data with the virtual space's coordinate system and sets appropriate position information. This allows the user to smoothly browse the virtual space using a virtual reality device.

[0848] Step 6: User experience of the virtual space

[0849] Input: 3D data placed in virtual space

[0850] Output: User experience data (viewpoint movement, action data)

[0851] Specific operation: The user puts on the virtual reality device (e.g., VR goggles) and launches the dedicated application. They click the "Start Experience" button from the application's menu and move into the virtual space. The user walks around the virtual space, changes their viewpoint, and operates the controller. This experience data is recorded in real time.

[0852] Step 7: Shared experiences with multiple users

[0853] Input: User invitation request, location and viewpoint data of other users

[0854] Output: Synchronized experience data for multiple users

[0855] How it works: A user clicks the "Invite" button in the dedicated application, enters their friend's user ID, and sends an invitation link. The invited user also uses the virtual reality device and application to experience the same virtual space. The server synchronizes the location and viewpoint data of multiple users in real time, displays each user's avatar, and enables a shared experience.

[0856] Step 8: Emotion Recognition Engine in Action

[0857] Input: User's facial expression data and voice data

[0858] Output: Parsed emotion data

[0859] Specific operation: The server sends data collected from the facial recognition camera and microphone installed on the user's device to the emotion recognition engine. For example, the front camera is used to detect the user's facial expressions, and voice data is recorded by the microphone. The emotion recognition engine analyzes this data and determines the user's emotional state in real time.

[0860] Step 9: Change the virtual space based on emotion data

[0861] Input: Emotion data sent from the emotion recognition engine

[0862] Output: Dynamically changing virtual space objects and effects

[0863] Specific operation: The server dynamically changes the objects and effects in the virtual space based on the emotional data obtained from the emotion recognition engine. For example, if the user makes a surprised expression, the objects in the virtual space will all move in unison and special effects will be displayed.

[0864] Step 10: Emotion sharing among multiple users

[0865] Input: Emotion data from multiple users

[0866] Output: Changes in the virtual space based on shared emotional data

[0867] Specific operation: The server receives emotional data from multiple users in real time and shares it in the virtual space. For example, the emotional data of user A can be displayed to user B, allowing them to understand each other's emotional states. When user A smiles, user B's avatar also smiles. In this way, emotions can be shared among multiple users.

[0868] (Application example 2)

[0869] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0870] The present invention relates to a system that uses two-dimensional data to provide a three-dimensional experience and allows users to share that experience. However, conventional systems have the problem of being unable to dynamically change the environment or objects in the virtual space in response to the user's emotions, resulting in a lack of immersion and emotional sharing between users. Furthermore, when multiple users share an experience in a virtual reality space, the lack of real-time sharing of emotional data and changes to the environment based on those emotions limits the experience.

[0871] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for a user to upload 2D data from a terminal, means for receiving the 2D data and applying an artificial intelligence model to convert the 2D data into 3D data, means for arranging the converted 3D data in a virtual space, means for a user to experience the 3D data in the virtual space via a virtual reality device, means for multiple users to share and experience the 3D data in the same virtual space, means for recognizing a user's emotion and dynamically changing objects and effects in the virtual space based on the emotion, and means for sharing emotion data of multiple users in real time and dynamically changing effects in the virtual space based on the data. As a result, the environment and objects in the virtual space dynamically change according to the user's emotion, enabling emotions to be shared among multiple users.

[0872] "Two-dimensional data" is digital data that is represented on a plane, such as photographs, images, and videos.

[0873] "3D data" is data that has depth and a sense of three-dimensionality, and is digital data that includes meshes and textures.

[0874] An "artificial intelligence model" is an algorithm that uses machine learning and deep learning, and is a computational tool for analyzing two-dimensional data and converting it into three-dimensional data.

[0875] A "virtual space" is a computer-generated, three-dimensional digital environment that a user can experience through a virtual reality device.

[0876] A "virtual reality device" is a device that allows users to experience data in a virtual space, such as a VR headset or HMD (Head-Mounted Display).

[0877] An "emotion recognition engine" is software or hardware that analyzes a user's facial expressions and voice to determine their emotional state.

[0878] "Effects" are visual or auditory effects displayed within the virtual space, which are dynamically generated according to the user's emotional state.

[0879] A "3D mesh" is a polygon structure formed by vertices, edges, and faces that make up a three-dimensional shape, and is a data structure that provides an actual sense of three-dimensionality.

[0880] A "texture" is image data that is applied to a 3D mesh to realistically represent the texture and color of an object's surface.

[0881] An "avatar" is a digital character that represents a user in a virtual space.

[0882] "Emotion data" is data that indicates the emotional state of the user analyzed by an emotion recognition engine.

[0883] "Real-time synchronization" means that the location information, viewpoint data, emotional data, etc. of multiple users in a virtual space are updated simultaneously, allowing all users to share the latest information.

[0884] The present invention is a system that allows users to experience 2D data in 3D within a virtual space, share that experience among multiple users, and dynamically change objects and effects within the virtual space based on the user's emotions. The realization of this system involves the following means and processes.

[0885] First, users use their devices to upload two-dimensional data, which refers to digital data represented on a flat surface, such as photographs, images, and videos, which are selected and uploaded through the device's user interface.

[0886] The server then receives the uploaded 2D data, verifies the data, and then converts it into 3D data using a generative AI model. A generative AI model is an algorithm designed using machine learning or deep learning that analyzes 2D image data, estimates 3D shapes, and generates 3D meshes and textures based on the 3D shapes.

[0887] The server then places the generated 3D data in a virtual space, which is a computer-generated 3D digital environment that users can experience through a virtual reality device. Users can wear the device and view and experience the 3D data in the virtual space.

[0888] Another feature of the present invention is an emotion recognition engine that recognizes a user's emotions in real time. An emotion recognition engine is software or hardware for analyzing a user's emotional state from their facial expressions and voice. The server collects the user's facial expressions and voice and sends them to the emotion recognition engine. The emotion recognition engine analyzes this data and determines the user's emotions.

[0889] Based on the emotional data, the server dynamically changes the objects and effects in the virtual space. For example, if the user expresses surprise, objects in the virtual space may suddenly move or visual effects may be displayed. This makes the user's experience more intuitive and immersive.

[0890] Furthermore, the system includes a means for sharing emotional data among multiple users in real time. When multiple users are experiencing the same virtual space, they can share their emotional data and dynamically change the effects in the virtual space based on that data. This allows multiple users to share a common experience and feel a sense of emotional unity.

[0891] A specific use case is a scenario where a user uploads a video of a family trip, converts it into 3D, and relives it using a virtual reality device. When the emotion recognition engine recognizes the user's smile, effects such as fireworks appear in the virtual space. Friends can also be invited to the same virtual space to enjoy the experience together.

[0892] Example prompt sentence:

[0893] "Upload a video of your family trip"

[0894] "Displays various effects when the face is surprised"

[0895] "Invite friends to join you in VR"

[0896] This allows users to enjoy a richer, more shareable virtual reality experience.

[0897] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0898] Step 1:

[0899] The user uploads 2D data using the terminal. Specifically, the user selects a 2D data file such as a photo or video and clicks the upload button, and the terminal sends the selected data to the server. The input is the 2D data file, and the output is the data sent to the server.

[0900] Step 2:

[0901] The server receives the 2D data. The server verifies the format and file size of the received data to ensure it is valid. The input is the 2D data file sent from the terminal, and the output is the verified data.

[0902] Step 3:

[0903] The server sends the verified 2D data to the generative AI model, which uses machine learning algorithms to analyze the 2D data and convert it into 3D data. The input is a 2D data file, and the output is 3D data.

[0904] Step 4:

[0905] The server places the generated 3D data in a virtual space. The server then places the 3D mesh and texture in the appropriate position within the virtual space, allowing the user to experience it using a virtual reality device. The input is the 3D data output by the generative AI model, and the output is 3D data placed within the virtual space.

[0906] Step 5:

[0907] The user wears a virtual reality device and experiences 3D data in a virtual space. The user puts on a VR headset and launches a dedicated application to enter the virtual space. The input is 3D data placed in the virtual space, and the output is the user's experience.

[0908] Step 6:

[0909] The server sends the user's facial expressions and voice to the emotion recognition engine. The camera and microphone installed in the virtual reality device capture the user's facial expressions and voice in real time and send them to the server. The server then sends them to the emotion recognition engine. The input is the user's facial expression and voice data, and the output is the input data for the emotion recognition engine.

[0910] Step 7:

[0911] The emotion recognition engine determines the user's emotional state. The emotion recognition engine analyzes facial and voice data to determine the user's emotion. The input is the captured facial and voice data, and the output is the user's emotional state.

[0912] Step 8:

[0913] The server dynamically changes objects and effects in the virtual space based on the emotional data. For example, if the user makes a surprised expression, a specific effect is displayed. The input is the user's emotional data, and the output is the changed effects and objects in the virtual space.

[0914] Step 9:

[0915] Emotional data from multiple users is shared in real time. The server receives each user's emotional data and synchronizes it within the virtual space. This allows the emotions to be shared with other users, and corresponding effects are activated. The input is the emotional data of multiple users, and the output is the emotional data synchronized in real time and effects based on it.

[0916] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0917] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0918] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0919] [Third embodiment]

[0920] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0921] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0922] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0923] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0924] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0925] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0926] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0927] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0928] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0929] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0930] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0931] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0932] The present invention is a system that allows users to experience 2D data in 3D in a virtual space and share that experience among multiple users. This system mainly consists of the following steps.

[0933] 1. The user uploads 2D data

[0934] Users use their own devices to select and upload 2D data such as photos and images from their local disks, which the devices then receive and send to the server.

[0935] 2. The server receives the 2D data and sends it to the AI ​​model.

[0936] The server receives the uploaded 2D data, verifies the data format and file size, and then sends it to an AI model, which uses machine learning algorithms to analyze the 2D data and convert it into 3D data.

[0937] 3. Artificial intelligence models transform 2D data into 3D

[0938] The AI ​​model uses image analysis technology to identify the shape and depth of people and objects from 2D data. Based on this information, it generates a 3D mesh and applies textures to it, resulting in realistic 3D data.

[0939] 4. The server places the 3D data in the virtual space

[0940] The generated 3D data is placed in a virtual space by the server, and the space is ready for the user to view using a virtual reality device.

[0941] 5. The user experiences the virtual space using a virtual reality device.

[0942] Users wear virtual reality devices and launch a dedicated application to view and experience 3D data placed in a virtual space. Users can change their viewpoint and walk around the virtual space.

[0943] 6. Shared experiences among multiple users

[0944] Users can invite other users to the virtual space. Invited users can also use the same virtual reality device and application to experience the same virtual space. The server synchronizes the location and viewpoint data of multiple users in real time and displays each user's avatar, enabling a shared experience.

[0945] Specific examples

[0946] As a specific example, a case where photos of a family trip are used will be described.

[0947] User A: Uploads a group photo of a family trip from the device. The photo is a file called "familytrip.jpg." User A operates the device, selects the photo, and clicks the "Upload" button.

[0948] Server: Receives the upload request, saves "Family Trip.jpg", and sends it to the AI ​​model, which analyzes the photo, identifies the positions of family faces and the background scenery, and generates a 3D mesh.

[0949] Artificial intelligence model: Generates a 3D mesh and applies texture based on color information extracted from the original photo.

[0950] Server: Uploads the generated 3D data to the virtual space and links it to User A's profile.

[0951] User A: Puts on the VR goggles, launches the Metaverse application, and selects the "Family Trip" album. User A relives the family trip in a virtual space in a realistic way.

[0952] User A: Furthermore, User B invites Friend B to this virtual space. The server grants Friend B permission to view the space, and Friend B also puts on VR goggles and joins the virtual space. The two can relive memories of their family trip together and communicate in real time.

[0953] This system allows users to not only memorize memorable scenes as photographs, but also recreate them in three dimensions, enabling shared experiences in real time.

[0954] The processing flow will be explained below.

[0955] Step 1:

[0956] The user uses the terminal to select two-dimensional data (e.g., a photo file).

[0957] The user opens a browser or a dedicated application, accesses the photo upload screen, selects a file such as "Family Trip.jpg" from the local disk, and clicks the upload button.

[0958] Step 2:

[0959] The device sends the selected two-dimensional data to the server.

[0960] The device generates an HTTP POST request and sends the selected photo data and user ID to the server, along with metadata such as file name, file format, and file size.

[0961] Step 3:

[0962] The server receives and verifies the 2D data.

[0963] The server receives the HTTP POST request, saves the photo data in temporary storage, and performs validation checks such as checking for valid file formats and file sizes, and virus checks.

[0964] Step 4:

[0965] The server sends the photo data to an artificial intelligence model.

[0966] Once verified, the photo data is passed to the AI ​​model, which then sends the photo data and associated metadata to the AI ​​model.

[0967] Step 5:

[0968] An artificial intelligence model analyzes the two-dimensional data.

[0969] The AI ​​model identifies people and objects in the photo, analyzes their shape and depth, and uses deep learning algorithms to analyze each pixel in the photo and generate mesh data to recreate the three-dimensional structure.

[0970] Step 6:

[0971] An artificial intelligence model generates 3D meshes and textures.

[0972] A three-dimensional shape is created based on the mesh data. Color information is extracted from the original photo and texture is applied to the generated mesh. This results in a 3D model that looks realistic.

[0973] Step 7:

[0974] The server places the generated 3D data in a virtual space.

[0975] The completed 3D data is placed at a specific location in the virtual space. If multiple 3D objects are required, each object is placed in the appropriate location.

[0976] Step 8:

[0977] The server links the 3D data to the user's profile.

[0978] The generated 3D data ID is linked to the user ID and saved in a database, allowing the user to access the 3D data at any time.

[0979] Step 9:

[0980] The user puts on the virtual reality device and starts a dedicated application.

[0981] The user puts on the VR goggles and starts the Metaverse application. After the user is authenticated, a list of available albums is displayed within the application.

[0982] Step 10:

[0983] The user selects and views the generated 3D album.

[0984] Select "Family Trip" from the album list and view the 3D data placed in the virtual space. Users can change the viewpoint and walk around the virtual space.

[0985] Step 11:

[0986] A user sends a request to the server to invite others.

[0987] A user generates a request on their device to invite other users to the same virtual space and sends it to the server, which generates an invitation link or sharing code.

[0988] Step 12:

[0989] The server adds viewing permissions to other people's profiles.

[0990] The server receives the invitation request and adds the permission to view the three-dimensional data to the profile of the other specified user.

[0991] Step 13:

[0992] Other users access the virtual space using the invitation link.

[0993] Other users click on the invitation link and join the same virtual space using their virtual reality devices. The server synchronizes data in real time, and multiple avatars are displayed simultaneously.

[0994] Step 14:

[0995] The server synchronizes the location information and viewpoint data among multiple users.

[0996] The server receives each user's position and viewpoint data in real time and transmits that information to other users, allowing each user to see the avatars of other users in real time.

[0997] This series of steps allows users to re-experience a simple two-dimensional photo in a realistic three-dimensional space and share it with other users.

[0998] Example 1

[0999] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1000] Modern virtual reality technology is often experienced by users alone, and individual experiences tend to be fragmented. Furthermore, the process of converting 2D data into 3D and placing it in a virtual space is extremely complex and requires extensive specialized knowledge. As a result, it is difficult for ordinary users to create realistic 3D experiences from 2D data and share them with other users. Furthermore, it is difficult for multiple users to share an experience in real time in the same virtual space, limiting collaborative experiences.

[1001] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1002] In this invention, the server includes a means for users to upload 2D data from their devices, a means for receiving the 2D data and applying a generative AI model to convert it into 3D data, and a means for placing the converted 3D data in a virtual space. This allows general users to easily convert 2D data into 3D data, create realistic virtual experiences, and share them with other users. It also allows multiple users to experience 3D data in real time within the same virtual space, synchronizing each user's location information and viewpoint data. As a result, users can enjoy a collaborative virtual experience, enabling virtual reality technology to be enjoyed on a wider scale.

[1003] A "user" is a person who uses a terminal to upload two-dimensional data and experience virtual reality.

[1004] A "terminal" refers to an electronic device used by a user, such as a computer or smartphone.

[1005] "Two-dimensional data" refers to two-dimensional information such as photos and image data uploaded by users.

[1006] "Upload" refers to the act of a user sending data from a terminal to a server.

[1007] A "server" is a computer system that processes two-dimensional data received from users, converts it into three-dimensional data, and places it in a virtual space.

[1008] A "generative AI model" is a program that uses artificial intelligence technology to analyze two-dimensional data and convert it into three-dimensional data.

[1009] "3D data" refers to three-dimensional data converted from 2D data by a generative AI model.

[1010] "Virtual space" refers to a three-dimensional virtual space generated by computer graphics.

[1011] A "virtual reality device" refers to a head-mounted display or VR goggles that users use to experience a virtual space.

[1012] A "shared experience" is when multiple users simultaneously experience virtual reality in the same virtual space.

[1013] "Location information" refers to coordinate data within the user's virtual space.

[1014] "Viewpoint data" refers to data that indicates the visual direction and angle of the user within the virtual space.

[1015] An "invitation request" is a request made by a user to invite another user to a virtual space.

[1016] "Viewing authority" refers to the authority that allows an invited user to view data in a virtual space.

[1017] An "avatar" is a three-dimensional character that represents a user in a virtual space.

[1018] "Real-time synchronization" refers to the simultaneous updating of location information and viewpoint data for multiple users within a virtual space.

[1019] The present invention is a system that allows users to upload 2D data from their devices, convert it into 3D data, and share it in a virtual space. The steps for implementing this system are as follows:

[1020] First, the user uploads 2D data using a terminal. The terminal can be an electronic device such as a PC or smartphone. The user opens a file selection dialog, selects a photo or image data, such as "Family Trip.jpg," from the local disk, and clicks the upload button to send the data to the server. The technology used is an HTTP POST request.

[1021] Next, the server receives the 2D data. Using a web framework such as Flask or Django, the server stores the data appropriately and validates the data format and file size. The server then sends the 2D data to the artificial intelligence model for generative AI model application. The server uses a REST API or JSON format for this purpose.

[1022] The generative AI model uses machine learning libraries such as TensorFlow and PyTorch to analyze 2D data and convert it into 3D data. The AI ​​model analyzes the received 2D data to extract object contours and estimate depth. Based on the results of this analysis, it generates a 3D mesh and applies texture using color information extracted from the original image.

[1023] The server receives the generated 3D data and places it in a virtual space. The virtual space is built using a virtual space engine such as Unity or Unreal Engine. The server stores the generated 3D data in a database and places it appropriately in the virtual space.

[1024] Users wear virtual reality devices (such as Oculus Rift or HTC Vive) and view and experience 3D data in a virtual space. By launching a dedicated application, users can move freely within the virtual space and interact with it.

[1025] In addition, multiple users can share the experience of 3D data in the same virtual space. Users can invite other users to the virtual space, and the invited users also use virtual reality devices. The server synchronizes the location and viewpoint data of multiple users in real time and displays the avatars of other users.

[1026] Specific examples

[1027] As a concrete example, let's consider the case of a family trip photo. User A uploads a group photo of the family trip, "familytrip.jpg," from their device. The server receives the upload request, saves "familytrip.jpg," and sends it to the generative AI model. The generative AI model analyzes the photo, identifies the position of family members' faces and the background scenery, and generates a 3D mesh. The server uploads the generated 3D data to the virtual space and links it to User A's profile. User A puts on VR goggles, launches the Metaverse application, and selects the "Family Trip" album. Within the virtual space, they relive their family trip memories in a way that is close to reality. User A invites Friend B to this virtual space, and the server grants Friend B viewing permissions as well. Friend B also puts on VR goggles and joins the virtual space, and the two can experience their family trip memories together and communicate in real time.

[1028] Prompt Sentence Examples

[1029] What are the steps to upload photos from a family trip, turn them into 3D, experience them in a virtual space using a VR device, and share them with friends?

[1030] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1031] Step 1:

[1032] The user selects an image on the device

[1033] The user opens a file selection dialog using the terminal and selects an image to upload from the local disk (for example, "Family Trip.jpg"). The input is the 2D data (image file) selected by the user. The output is the path information of the selected image file.

[1034] Step 2:

[1035] The device sends the data to the server

[1036] When the user clicks the "Upload" button, the device sends the selected image data to the server as an HTTP POST request. Specifically, it uses the JavaScript fetch API to send the image data as form data. The input is the path information of the selected image file, and the output is the image data sent to the server.

[1037] Step 3:

[1038] The server receives the data

[1039] The server receives the image data sent from the device. It processes the HTTP POST request using a web framework such as Flask or Django. Specifically, the server temporarily stores the image data and verifies the format and size. The input is the image data in the HTTP request, and the output is the verified image data.

[1040] Step 4:

[1041] The server sends the data to the artificial intelligence model

[1042] The server sends the received image data to the generative AI model. Specifically, it uses a REST API to send the image data in JSON format. The input is the verified image data, and the output is an analysis request to the artificial intelligence model.

[1043] Step 5:

[1044] An artificial intelligence model analyzes the data

[1045] The generative AI model uses TensorFlow and PyTorch to perform image analysis. Specifically, it performs object recognition and depth estimation in the image to generate a 3D mesh. The input is 2D image data sent from the server, and the output is 3D mesh data.

[1046] Step 6:

[1047] Applying textures to 3D meshes

[1048] The AI ​​model applies texture to the generated 3D mesh using color information extracted from the original image. Specifically, it uses texture mapping technology. The input is 3D mesh data and texture information, and the output is the completed 3D data.

[1049] Step 7:

[1050] The server places the 3D data in the virtual space

[1051] The server receives the generated 3D data and places it in a virtual space created with Unity or Unreal Engine. Specifically, it stores the data in a database and transfers it to the virtual space engine. The input is the completed 3D data, and the output is 3D data placed in the virtual space.

[1052] Step 8:

[1053] Users experience a virtual space using a virtual reality device.

[1054] The user puts on the VR goggles and launches a dedicated application. Specific actions include visually viewing and manipulating objects in a virtual space. The input is 3D data placed in the virtual space, and the output is the visual information and interaction information of the virtual reality that the user experiences.

[1055] Step 9:

[1056] Shared experiences among multiple users

[1057] Users can invite other users into the virtual space, and the server synchronizes each user's location information and viewpoint data in real time. Specifically, the invited users also wear the same virtual reality device and use real-time communication technology. The input is the location information and viewpoint data of multiple users, and the output is a real-time shared virtual space experience.

[1058] Prompt Sentence Examples

[1059] What are the steps to upload photos from a family trip, turn them into 3D, experience them in a virtual space using a VR device, and share them with friends?

[1060] (Application example 1)

[1061] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1062] The present invention relates to a system that not only allows users to experience two-dimensional data in three dimensions within a virtual space, but also allows multiple users to share that experience in real time. In particular, the objective of the present invention is to provide a system that allows users to enjoy product experiences in a virtual store in three dimensions and share shopping experiences with other users.

[1063] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1064] In this invention, the server includes means for users to upload 2D data from their terminals, means for receiving the 2D data and applying an artificial intelligence model to convert it into 3D data, means for arranging the converted 3D data in a virtual space, means for users to experience the 3D data in the virtual space through a virtual reality device, means for multiple users to share the experience of the 3D data in the same virtual space, means for users to upload product images and view and experience products in a 3D manner in a virtual store, and means for inviting other users to the virtual store and sharing the shopping experience in real time, thereby enabling users to experience products in a 3D manner in the virtual store and share that experience with other users.

[1065] A "terminal" refers to an electronic device used by a user, such as a computer, smartphone, or tablet.

[1066] "Two-dimensional data" refers to digital data that contains two-dimensional information in terms of length and width, such as images and photographs.

[1067] "Upload" refers to the action of sending data from a user's terminal to a server.

[1068] "3D data" refers to three-dimensional digital data that has depth.

[1069] An "artificial intelligence model" is a data analysis technology that uses machine learning algorithms and has the ability to convert two-dimensional data into three-dimensional data.

[1070] A "virtual space" is a virtual environment that differs from reality and is constructed on a computer using digital technology.

[1071] A "virtual reality device" is a device that allows users to have an experience similar to the real world in a digital space, such as VR goggles or a head-mounted display.

[1072] "Product images" refer to photographs or digital images of products used in online shopping, etc.

[1073] A "virtual store" is a virtual shop that exists on the Internet where you can browse and purchase products.

[1074] "Viewing in 3D" refers to the act of using 3D data to display depth information, providing an experience that feels as if you are looking at the real thing.

[1075] A "shopping experience" is a series of online activities in which a user browses products and considers a purchase.

[1076] A "shared experience" refers to multiple users experiencing the same experience in real time in the same virtual space.

[1077] "Real-time" refers to processing and information exchange occurring immediately, without delay.

[1078] System Overview

[1079] The present invention relates to a system that allows users to experience two-dimensional data in a three-dimensional virtual space and share that experience among multiple users. In particular, the present invention provides a system that allows users to view products in a virtual store in three dimensions and share their shopping experience with other users.

[1080] Hardware and Software Used

[1081] Server: Use a high performance server, e.g. AWS EC2.

[1082] Virtual reality devices: Users use VR goggles or head-mounted displays. Example: Oculus Rift.

[1083] Devices: Smartphones, smart glasses, head-mounted displays, etc. Examples: Google Glass, HoloLens.

[1084] Machine learning libraries: TensorFlow, PyTorch.

[1085] 3D modeling software: Blender, Unity engine.

[1086] Details of data processing and calculation

[1087] 1. Data upload

[1088] The user uploads a product image using their device. For example, they send "New Product Bag.jpg" from their smartphone to the server. The API used for this is the REST API.

[1089] 2. Image analysis and 3D conversion

[1090] The server analyzes the received product images using machine learning algorithms. It uses a convolutional neural network (CNN)-based model to identify the shape and depth of the object in the image, then generates a 3D mesh and applies textures to it.

[1091] 3. Placement in virtual space

[1092] The server places the generated 3D data in the virtual store using the Unity engine, which optimally positions the 3D model in the virtual space.

[1093] 4. Virtual Experiences

[1094] Users wear VR goggles and experience a virtual store through the application. The server synchronizes the user's viewpoint data and location information in real time, allowing users to freely walk around the virtual store and experience the products in three dimensions.

[1095] 5. Shared Experiences

[1096] Users can invite other users to the virtual store. The server receives the invitation request and grants viewing privileges to other users. Location and viewpoint data are synchronized in real time between multiple users, allowing them to share their shopping experience.

[1097] Specific examples

[1098] User A uploads "New Product Bag.jpg" from their smartphone.

[1099] After receiving the image, the server converts it into a 3D bag using an artificial intelligence model and places it inside the virtual store.

[1100] User A puts on VR goggles and experiences the sensation of actually picking up a bag in a virtual store.

[1101] User A invites friend B to a virtual store and they shop together while viewing new products in real time.

[1102] Prompt Sentence Examples

[1103] Design an AI model that generates a 3D model from an image. The input is a product image file (e.g., "New Product Bag.jpg"), and the output is a 3D mesh and texture for display in a virtual space. The model must identify shape and depth from the image and generate a realistic 3D model. The 3D model must also be optimized for effective display in a virtual store.

[1104] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1105] Step 1:

[1106] The user uploads 2D data from their device. The input is a product image (e.g., "New Product Bag.jpg"), and the output is the image sent to the server. This data is sent to the server and received using a REST API.

[1107] Step 2:

[1108] The server validates the received 2D data. The input is the uploaded image file, and the output is the validation result to see if the data format and file size are correct. If the data is not in the correct format at this step, an error message is returned.

[1109] Step 3:

[1110] The server sends the data to an artificial intelligence model, which converts it into 3D data. The input is verified 2D data, and the output is 3D data with 3D mesh and texture. The model used here is a Convolutional Neural Network (CNN)-based analysis algorithm that identifies shape and depth to generate a 3D model.

[1111] Step 4:

[1112] The server places the generated 3D data in a virtual space. The input is 3D data, and the output is 3D data placed at specific coordinates in the virtual space. The Unity engine is used to render the 3D model in the virtual space.

[1113] Step 5:

[1114] Users wear virtual reality devices and experience 3D data in a virtual space. The input is the user's viewpoint data and position information, and the output is an interactive experience in the virtual space. The server synchronizes this data in real time, allowing users to move freely within the virtual space.

[1115] Step 6:

[1116] Users can invite other users to a virtual space to share experiences. The input is an invitation request and the other user's ID, and the output is the granting of viewing permissions to the other users. Upon receiving the invitation, the server synchronizes the location and viewpoint data of multiple users in real time, providing a shared experience in the virtual space.

[1117] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1118] This invention is a system that allows users to experience 2D data in 3D in a virtual space and share that experience with multiple users, and also adds technology that recognizes the user's emotions and dynamically changes the objects and effects in the virtual space according to the emotions. This system mainly consists of the following steps.

[1119] 1. The user uploads 2D data

[1120] Users use their own devices to select and upload 2D data such as photos and images from their local disks, which the devices then receive and send to the server.

[1121] 2. The server receives the 2D data and sends it to the AI ​​model.

[1122] The server receives the uploaded 2D data, verifies the data format and file size, and then sends it to an AI model, which uses machine learning algorithms to analyze the 2D data and convert it into 3D data.

[1123] 3. Artificial intelligence models transform 2D data into 3D

[1124] The AI ​​model uses image analysis technology to identify the shape and depth of people and objects from 2D data. Based on this information, it generates a 3D mesh and applies textures to it, resulting in realistic 3D data.

[1125] 4. The server places the 3D data in the virtual space

[1126] The generated 3D data is placed in a virtual space by the server, and the space is ready for the user to view using a virtual reality device.

[1127] 5. The user experiences the virtual space using a virtual reality device.

[1128] Users wear virtual reality devices and launch a dedicated application to view and experience 3D data placed in a virtual space. Users can change their viewpoint and walk around the virtual space.

[1129] 6. Shared experiences among multiple users

[1130] Users can invite other users to the virtual space. Invited users can also use the same virtual reality device and application to experience the same virtual space. The server synchronizes the location and viewpoint data of multiple users in real time and displays each user's avatar, enabling a shared experience.

[1131] 7. Introducing an emotion recognition engine

[1132] The user's facial expression is captured using a camera and microphone, and the captured facial expressions and vocal tone are sent to an emotion recognition engine, which analyzes the data and determines the user's emotional state in real time.

[1133] 8. Changes in virtual space based on emotional data

[1134] Based on the recognized emotion data, objects and effects in the virtual space are dynamically changed. For example, if the user makes a surprised expression, all the objects in the virtual space will move in unison, or special effects will be displayed.

[1135] 9. Emotion Sharing among Multiple Users

[1136] When multiple users are in the same virtual space, they share their emotional data in real time. The server receives this emotional data and reflects it on each user's avatar and environment, making it easier for other users to see their emotional state.

[1137] Specific examples

[1138] As a specific example, a case where photos of a family trip are used will be described.

[1139] User A: Uploads a group photo of a family trip from the device. The photo is a file called "familytrip.jpg." User A operates the device, selects the photo, and clicks the "Upload" button.

[1140] Server: Receives the upload request, saves "Family Trip.jpg", and sends it to the AI ​​model, which analyzes the photo, identifies the positions of family faces and the background scenery, and generates a 3D mesh.

[1141] Artificial intelligence model: Generates a 3D mesh and applies texture based on color information extracted from the original photo.

[1142] Server: Uploads the generated 3D data to the virtual space and links it to User A's profile.

[1143] User A: Puts on the VR goggles, launches the Metaverse application, and selects the "Family Trip" album. User A relives the family trip in a virtual space in a realistic way.

[1144] Emotion recognition engine: Monitors user A's facial expressions and voice and analyzes their emotional state in real time. For example, if user A smiles, a character in the virtual space may greet them.

[1145] User A: Then, he invites Friend B to join the virtual space. The server grants Friend B permission to view the content, and Friend B also puts on VR goggles and joins the virtual space. The two can relive memories of their family trip together and share their emotions in real time. For example, if User A makes a surprised expression, that facial expression data is transmitted to Friend B, and a surprise effect may be displayed in the virtual space.

[1146] This system allows users to not only memorize memorable scenes as two-dimensional photographs, but also recreate them in three dimensions, allowing them to be experienced and shared in a virtual space that changes dynamically according to the user's emotions.

[1147] The processing flow will be explained below.

[1148] Step 1:

[1149] The user uses the terminal to select two-dimensional data (e.g., a photo file).

[1150] The user opens a browser or a dedicated application, accesses the photo upload screen, selects a file such as "Family Trip.jpg" from the local disk, and clicks the upload button.

[1151] Step 2:

[1152] The device sends the selected two-dimensional data to the server.

[1153] The device generates an HTTP POST request and sends the selected photo data and user ID to the server, along with metadata such as file name, file format, and file size.

[1154] Step 3:

[1155] The server receives and verifies the 2D data.

[1156] The server receives the HTTP POST request, saves the photo data in temporary storage, and performs validation checks such as checking for valid file formats and file sizes, and virus checks.

[1157] Step 4:

[1158] The server sends the photo data to an artificial intelligence model.

[1159] Once verified, the photo data is passed to the AI ​​model, which then sends the photo data and associated metadata to the AI ​​model.

[1160] Step 5:

[1161] An artificial intelligence model analyzes the two-dimensional data.

[1162] The AI ​​model identifies people and objects in the photo, analyzes their shape and depth, and uses deep learning algorithms to analyze each pixel in the photo and generate mesh data to recreate the three-dimensional structure.

[1163] Step 6:

[1164] An artificial intelligence model generates 3D meshes and textures.

[1165] A three-dimensional shape is created based on the mesh data. Color information is extracted from the original photo and texture is applied to the generated mesh. This results in a 3D model that looks realistic.

[1166] Step 7:

[1167] The server places the generated 3D data in a virtual space.

[1168] The completed 3D data is placed at a specific location in the virtual space. If multiple 3D objects are required, each object is placed in the appropriate location.

[1169] Step 8:

[1170] The server links the 3D data to the user's profile.

[1171] The generated 3D data ID is linked to the user ID and saved in a database, allowing the user to access the 3D data at any time.

[1172] Step 9:

[1173] The user puts on the virtual reality device and starts a dedicated application.

[1174] The user puts on the VR goggles and starts the Metaverse application. After the user is authenticated, a list of available albums is displayed within the application.

[1175] Step 10:

[1176] The user selects and views the generated 3D album.

[1177] Select "Family Trip" from the album list and view the 3D data placed in the virtual space. Users can change the viewpoint and walk around the virtual space.

[1178] Step 11:

[1179] A user sends a request to the server to invite others.

[1180] A user generates a request on their device to invite other users to the same virtual space and sends it to the server, which generates an invitation link or sharing code.

[1181] Step 12:

[1182] The server adds viewing permissions to other people's profiles.

[1183] The server receives the invitation request and adds the permission to view the three-dimensional data to the profile of the other specified user.

[1184] Step 13:

[1185] Other users access the virtual space using the invitation link.

[1186] Other users click on the invitation link and join the same virtual space using their virtual reality devices. The server synchronizes data in real time, and multiple avatars are displayed simultaneously.

[1187] Step 14:

[1188] The server synchronizes the location information and viewpoint data among multiple users.

[1189] The server receives each user's position and viewpoint data in real time and transmits that information to other users, allowing each user to see the avatars of other users in real time.

[1190] Step 15:

[1191] Users share experiences with other users in a virtual space.

[1192] Users can communicate with other users in a virtual space while viewing and experiencing the same 3D data.

[1193] Step 16:

[1194] The emotion recognition engine collects the user's facial expressions and voice data.

[1195] The emotion recognition engine uses the camera and microphone installed in the VR goggles to capture the user's facial expressions and voice tone, thereby grasping the user's real-time emotional state.

[1196] Step 17:

[1197] An emotion recognition engine analyzes the user's emotions.

[1198] The emotion recognition engine analyzes the collected data and determines what emotions the user is feeling, such as smiling, surprised, or sad.

[1199] Step 18:

[1200] The server reflects the recognized emotions in the virtual space.

[1201] When the user's emotion is recognized, the server applies that data to the virtual space, causing objects and effects in the virtual space to change dynamically. For example, if the user smiles, an effect that brightens the scenery is applied.

[1202] Step 19:

[1203] The server shares emotion data among multiple users.

[1204] When multiple users exist in a virtual space, the server also synchronizes each user's emotional data in real time, allowing each user to see the emotional state of the other users.

[1205] Step 20:

[1206] Other users experience the shared emotion data.

[1207] Other users in the virtual space can experience the emotional changes in real time and adjust their own actions and reactions accordingly.

[1208] This series of steps allows users to re-experience a simple two-dimensional photograph in a realistic three-dimensional space, and share the experience with other users in a virtual space that changes dynamically with changes in emotions.

[1209] Example 2

[1210] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1211] While existing technologies exist for systems that allow users to experience 2D real-world data (such as photos and images) in 3D virtual space, they lack the ability to recognize users' emotions in real time and dynamically change objects and effects in the virtual space accordingly, as well as the ability to share emotions in real time among multiple users. For this reason, there is a demand for systems that can make user experiences richer and more interactive.

[1212] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for a user to upload 2D data from a terminal, means for receiving the 2D data and verifying the data format and file size before transmitting it to an artificial intelligence model, means for analyzing the 2D data using the artificial intelligence model and converting it into 3D data, means for arranging the converted 3D data in a virtual space, means for a user to access the 3D data in the virtual space via a virtual reality device to view and experience it, means for multiple users to share and experience the 3D data in the same virtual space, means for collecting emotion data using a user's facial expression recognition camera or microphone and transmitting it to an emotion recognition engine, means for dynamically changing objects and effects in the virtual space based on the recognized emotion data, and means for sharing and reflecting emotion data with other users in real time. This enables dynamic changes in the virtual space according to the user's emotions and emotion sharing among multiple users.

[1213] A "terminal" is an electronic device used by a user, and includes forms such as a personal computer, a smartphone, and a tablet.

[1214] "Two-dimensional data" refers to data that exists on a plane, such as photographs, images, and graphics.

[1215] A "server" is a computer system that communicates with multiple terminals via a network and stores, processes, sends and receives data.

[1216] "Data format" refers to the method or structure in which digital data is stored, including image formats such as JPEG and PNG.

[1217] "File size" is an indicator of the size of data and is expressed in units such as kilobytes (KB) or megabytes (MB).

[1218] An "artificial intelligence model" refers to a software system that uses machine learning algorithms or deep learning to perform specific tasks.

[1219] "Image analysis" is the process of extracting and identifying features and patterns in two-dimensional data, and primarily uses deep learning technology.

[1220] "Three-dimensional data" is data that has depth information, including the shape and position of a three-dimensional object.

[1221] A "3D mesh" is a data structure that represents the surface of a 3D object and is composed of a large number of polygons (mainly triangles).

[1222] "Texture" refers to image or color information applied to a 3D mesh to make the object appear more realistic.

[1223] A "virtual space" is a computer-generated three-dimensional environment that users can visually experience.

[1224] "Virtual reality devices" refer to devices for experiencing virtual spaces, such as VR goggles and head-mounted displays (HMDs).

[1225] An "expression recognition camera" is a camera device that detects a user's facial expressions and captures them as data.

[1226] A "microphone" is a device for collecting sound and plays a role in transmitting the user's voice to the emotion recognition engine.

[1227] An "emotion recognition engine" is a software system that analyzes data such as facial expressions and tone of voice to recognize a user's emotional state.

[1228] "Emotional data" is data that represents the user's emotional state and is collected and analyzed in real time.

[1229] An "object" refers to any three-dimensional model or entity that exists in a virtual space.

[1230] "Effects" are visual effects that are displayed in response to specific events or actions within the virtual space.

[1231] "Location information" refers to the coordinate information of a user or avatar within a virtual space.

[1232] "Viewpoint data" is data that represents the user's line of sight and the camera direction.

[1233] This invention is a system that allows users to experience two-dimensional data in three dimensions within a virtual space and share that experience among multiple users, and also adds technology that recognizes the user's emotions and dynamically changes the objects and effects within the virtual space in response to those emotions.

[1234] An embodiment of this system includes the following process.

[1235] 1. User uploads 2D data

[1236] Users use their own devices (e.g., personal computers, smartphones, tablets) to select and upload 2D data such as photos and images from their local disks. Specifically, users click the file selection button and select the image file they want to upload. After making their selection, they press the "Upload" button, which causes the device to send a request to the server.

[1237] 2. Receipt and verification of data by the server

[1238] The server receives the upload request sent from the device. Then, it verifies the file format and size of the received 2D data. For example, if the format is invalid or the file size is too large, it returns an error message to the user.

[1239] 3. Sending data from the server to the AI ​​model

[1240] The server then sends the verified 2D data to the AI ​​model, preprocessing the data (e.g., resizing the image or converting the format). After sending, the server records the progress and may notify the user in real time.

[1241] 4. Analyzing and transforming data using artificial intelligence models

[1242] The AI ​​model uses image analysis techniques to identify the shape and depth of people and objects from 2D data. Specifically, it uses deep learning algorithms (e.g., convolutional neural networks) to extract feature points within the image. It then generates a 3D mesh based on these feature points and applies a texture based on color information extracted from the original photo.

[1243] 5. Placement of 3D data in virtual space by the server

[1244] The server places the 3D data received from the AI ​​model in the virtual space. Specifically, it places the data according to the virtual space's coordinate system and sets appropriate position information, allowing users to smoothly browse the virtual space using a virtual reality device.

[1245] 6. User Experience of Virtual Space

[1246] The user puts on the virtual reality device (e.g., VR goggles, head-mounted display) and launches a dedicated application. Specifically, they click the "Start Experience" button from the menu within the application and move into the virtual space. The user can change their viewpoint and walk around the virtual space using a controller.

[1247] 7. Shared experiences among multiple users

[1248] Users can invite other users to the virtual space by clicking the "Invite" button in the dedicated application, entering their friend's user ID, and sending an invitation link. Invited users can also use virtual reality devices and the application to experience the same virtual space. The server synchronizes the location and viewpoint data of multiple users in real time, and displays each user's avatar, enabling a shared experience.

[1249] 8. How the Emotion Recognition Engine Works

[1250] The server sends data collected from the facial recognition camera and microphone installed on the user's device to the emotion recognition engine. For example, the device uses the front camera to detect the user's facial expressions. In the case of audio data, the audio recorded by the microphone is sent. The emotion recognition engine analyzes this data and determines the user's emotional state in real time.

[1251] 9. Changes in virtual space based on emotional data

[1252] The server dynamically changes the objects and effects in the virtual space based on the emotional data obtained from the emotion recognition engine. If the user makes a surprised expression, the objects in the virtual space will move all at once, or special effects (e.g., fireworks or light effects) will be displayed.

[1253] 10. Emotion Sharing among Multiple Users

[1254] The server receives emotional data from multiple users in real time and shares it within the virtual space. User A's emotional data is also displayed to User B, allowing them to mutually confirm their emotional state. For example, if User A smiles, User B's avatar will also smile.

[1255] Reproducing specific examples

[1256] As a concrete example, we will explain the case where a photo of a family trip is used. User A uploads a group photo of the family trip, "familytrip.jpg", from his / her terminal. The detailed procedure is explained below.

[1257] Upload steps:

[1258] User A operates the terminal to select the family trip photo "familytrip.jpg" and clicks the "upload" button.

[1259] The server receives the upload request and sends it to the artificial intelligence model.

[1260] Data analysis and transformation:

[1261] An artificial intelligence model analyzes the photo, identifies the position of family members' faces and the background scenery, and generates a 3D mesh.

[1262] A texture is applied to the generated 3D mesh based on color information extracted from the original photo.

[1263] Placement in virtual space:

[1264] The server uploads the generated 3D data to the virtual space and links it to User A's profile.

[1265] Virtual space experience:

[1266] User A puts on the VR goggles, launches the Metaverse application, and selects the "Family Trip" album. In the virtual space, he or she relives memories of the family trip in a realistic way.

[1267] Emotion Recognition and Feedback:

[1268] The emotion recognition engine monitors the facial expressions and voice of user A and analyzes their emotional state in real time. For example, if user A smiles, a character in the virtual space may greet them.

[1269] Adding a shared experience:

[1270] User A invites Friend B to this virtual space, and the server grants Friend B viewing permission to join the virtual space. The two can relive memories of a family trip and share their emotions in real time. If User A makes a surprised expression, that facial expression data is transmitted to Friend B, and a surprise effect may be displayed in the virtual space.

[1271] Using this system, it is possible to realize an experience that goes beyond simple two-dimensional photographs, and instead involves a three-dimensional virtual space that changes dynamically according to the user's emotions.

[1272] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1273] Step 1: User uploads 2D data

[1274] Input: 2D data selected by the user (e.g., photo "Family Trip.jpg")

[1275] Output: Upload request from device to server

[1276] Specific operation: The user operates the device, clicks the file selection button, and selects 2D data such as a photo or image. After making a selection, the user presses the "Upload" button, which sends the selected data to the server. The device uploads this data as an HTTP request to the server in the specified format.

[1277] Step 2: Server receives and verifies data

[1278] Input: Upload request sent from the device (2D data)

[1279] Output: Validated 2D data or error message

[1280] Specific operation: After the server receives the upload request, it verifies the data format (e.g., JPEG, PNG) and file size (e.g., less than 2MB). If the format is invalid or the file size is too large, it returns an error message to the user. If the verification is successful, the received data proceeds to the next step.

[1281] Step 3: Sending data from the server to the AI ​​model

[1282] Input: Verified 2D data

[1283] Output: A request to send data to the AI ​​model

[1284] How it works: The server sends the verified 2D data to the AI ​​model. If necessary, it preprocesses the data (e.g., resizes or converts the image). The preprocessed data is sent to the AI ​​model and progress is recorded. The user may be notified of progress in real time.

[1285] Step 4: Analyze and transform the data with artificial intelligence models

[1286] Input: 2D data sent from the server

[1287] Output: 3D mesh, 3D data with texture information

[1288] How it works: The AI ​​model uses image analysis techniques to identify the shape and depth of people and objects from 2D data. It then uses deep learning algorithms (e.g., convolutional neural networks) to extract feature points within the image. This generates a 3D mesh and applies texture based on color information extracted from the original photo.

[1289] Step 5: The server places the 3D data in the virtual space.

[1290] Input: 3D data sent from an artificial intelligence model

[1291] Output: 3D data placed in virtual space

[1292] Specific operation: The server places the 3D data received from the AI ​​model in the virtual space. The server aligns the data with the virtual space's coordinate system and sets appropriate position information. This allows the user to smoothly browse the virtual space using a virtual reality device.

[1293] Step 6: User experience of the virtual space

[1294] Input: 3D data placed in virtual space

[1295] Output: User experience data (viewpoint movement, action data)

[1296] Specific operation: The user puts on the virtual reality device (e.g., VR goggles) and launches the dedicated application. They click the "Start Experience" button from the application's menu and move into the virtual space. The user walks around the virtual space, changes their viewpoint, and operates the controller. This experience data is recorded in real time.

[1297] Step 7: Shared experiences with multiple users

[1298] Input: User invitation request, location and viewpoint data of other users

[1299] Output: Synchronized experience data for multiple users

[1300] How it works: A user clicks the "Invite" button in the dedicated application, enters their friend's user ID, and sends an invitation link. The invited user also uses the virtual reality device and application to experience the same virtual space. The server synchronizes the location and viewpoint data of multiple users in real time, displays each user's avatar, and enables a shared experience.

[1301] Step 8: Emotion Recognition Engine in Action

[1302] Input: User's facial expression data and voice data

[1303] Output: Parsed emotion data

[1304] Specific operation: The server sends data collected from the facial recognition camera and microphone installed on the user's device to the emotion recognition engine. For example, the front camera is used to detect the user's facial expressions, and voice data is recorded by the microphone. The emotion recognition engine analyzes this data and determines the user's emotional state in real time.

[1305] Step 9: Change the virtual space based on emotion data

[1306] Input: Emotion data sent from the emotion recognition engine

[1307] Output: Dynamically changing virtual space objects and effects

[1308] Specific operation: The server dynamically changes the objects and effects in the virtual space based on the emotional data obtained from the emotion recognition engine. For example, if the user makes a surprised expression, the objects in the virtual space will all move in unison and special effects will be displayed.

[1309] Step 10: Emotion sharing among multiple users

[1310] Input: Emotion data from multiple users

[1311] Output: Changes in the virtual space based on shared emotional data

[1312] Specific operation: The server receives emotional data from multiple users in real time and shares it in the virtual space. For example, the emotional data of user A can be displayed to user B, allowing them to understand each other's emotional states. When user A smiles, user B's avatar also smiles. In this way, emotions can be shared among multiple users.

[1313] (Application example 2)

[1314] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1315] The present invention relates to a system that uses two-dimensional data to provide a three-dimensional experience and allows users to share that experience. However, conventional systems have the problem of being unable to dynamically change the environment or objects in the virtual space in response to the user's emotions, resulting in a lack of immersion and emotional sharing between users. Furthermore, when multiple users share an experience in a virtual reality space, the lack of real-time sharing of emotional data and changes to the environment based on those emotions limits the experience.

[1316] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for a user to upload 2D data from a terminal, means for receiving the 2D data and applying an artificial intelligence model to convert the 2D data into 3D data, means for arranging the converted 3D data in a virtual space, means for a user to experience the 3D data in the virtual space via a virtual reality device, means for multiple users to share and experience the 3D data in the same virtual space, means for recognizing a user's emotion and dynamically changing objects and effects in the virtual space based on the emotion, and means for sharing emotion data of multiple users in real time and dynamically changing effects in the virtual space based on the data. As a result, the environment and objects in the virtual space dynamically change according to the user's emotion, enabling emotions to be shared among multiple users.

[1317] "Two-dimensional data" is digital data that is represented on a plane, such as photographs, images, and videos.

[1318] "3D data" is data that has depth and a sense of three-dimensionality, and is digital data that includes meshes and textures.

[1319] An "artificial intelligence model" is an algorithm that uses machine learning and deep learning, and is a computational tool for analyzing two-dimensional data and converting it into three-dimensional data.

[1320] A "virtual space" is a computer-generated, three-dimensional digital environment that a user can experience through a virtual reality device.

[1321] A "virtual reality device" is a device that allows users to experience data in a virtual space, such as a VR headset or HMD (Head-Mounted Display).

[1322] An "emotion recognition engine" is software or hardware that analyzes a user's facial expressions and voice to determine their emotional state.

[1323] "Effects" are visual or auditory effects displayed within the virtual space, which are dynamically generated according to the user's emotional state.

[1324] A "3D mesh" is a polygon structure formed by vertices, edges, and faces that make up a three-dimensional shape, and is a data structure that provides an actual sense of three-dimensionality.

[1325] A "texture" is image data that is applied to a 3D mesh to realistically represent the texture and color of an object's surface.

[1326] An "avatar" is a digital character that represents a user in a virtual space.

[1327] "Emotion data" is data that indicates the emotional state of the user analyzed by an emotion recognition engine.

[1328] "Real-time synchronization" means that the location information, viewpoint data, emotional data, etc. of multiple users in a virtual space are updated simultaneously, allowing all users to share the latest information.

[1329] The present invention is a system that allows users to experience 2D data in 3D within a virtual space, share that experience among multiple users, and dynamically change objects and effects within the virtual space based on the user's emotions. The realization of this system involves the following means and processes.

[1330] First, users use their devices to upload two-dimensional data, which refers to digital data represented on a flat surface, such as photographs, images, and videos, which are selected and uploaded through the device's user interface.

[1331] The server then receives the uploaded 2D data, verifies the data, and then converts it into 3D data using a generative AI model. A generative AI model is an algorithm designed using machine learning or deep learning that analyzes 2D image data, estimates 3D shapes, and generates 3D meshes and textures based on the 3D shapes.

[1332] The server then places the generated 3D data in a virtual space, which is a computer-generated 3D digital environment that users can experience through a virtual reality device. Users can wear the device and view and experience the 3D data in the virtual space.

[1333] Another feature of the present invention is an emotion recognition engine that recognizes a user's emotions in real time. An emotion recognition engine is software or hardware for analyzing a user's emotional state from their facial expressions and voice. The server collects the user's facial expressions and voice and sends them to the emotion recognition engine. The emotion recognition engine analyzes this data and determines the user's emotions.

[1334] Based on the emotional data, the server dynamically changes the objects and effects in the virtual space. For example, if the user expresses surprise, objects in the virtual space may suddenly move or visual effects may be displayed. This makes the user's experience more intuitive and immersive.

[1335] Furthermore, the system includes a means for sharing emotional data among multiple users in real time. When multiple users are experiencing the same virtual space, they can share their emotional data and dynamically change the effects in the virtual space based on that data. This allows multiple users to share a common experience and feel a sense of emotional unity.

[1336] A specific use case is a scenario where a user uploads a video of a family trip, converts it into 3D, and relives it using a virtual reality device. When the emotion recognition engine recognizes the user's smile, effects such as fireworks appear in the virtual space. Friends can also be invited to the same virtual space to enjoy the experience together.

[1337] Example prompt sentence:

[1338] "Upload a video of your family trip"

[1339] "Displays various effects when the face is surprised"

[1340] "Invite friends to join you in VR"

[1341] This allows users to enjoy a richer, more shareable virtual reality experience.

[1342] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1343] Step 1:

[1344] The user uploads 2D data using the terminal. Specifically, the user selects a 2D data file such as a photo or video and clicks the upload button, and the terminal sends the selected data to the server. The input is the 2D data file, and the output is the data sent to the server.

[1345] Step 2:

[1346] The server receives the 2D data. The server verifies the format and file size of the received data to ensure it is valid. The input is the 2D data file sent from the terminal, and the output is the verified data.

[1347] Step 3:

[1348] The server sends the verified 2D data to the generative AI model, which uses machine learning algorithms to analyze the 2D data and convert it into 3D data. The input is a 2D data file, and the output is 3D data.

[1349] Step 4:

[1350] The server places the generated 3D data in a virtual space. The server then places the 3D mesh and texture in the appropriate position within the virtual space, allowing the user to experience it using a virtual reality device. The input is the 3D data output by the generative AI model, and the output is 3D data placed within the virtual space.

[1351] Step 5:

[1352] The user wears a virtual reality device and experiences 3D data in a virtual space. The user puts on a VR headset and launches a dedicated application to enter the virtual space. The input is 3D data placed in the virtual space, and the output is the user's experience.

[1353] Step 6:

[1354] The server sends the user's facial expressions and voice to the emotion recognition engine. The camera and microphone installed in the virtual reality device capture the user's facial expressions and voice in real time and send them to the server. The server then sends them to the emotion recognition engine. The input is the user's facial expression and voice data, and the output is the input data for the emotion recognition engine.

[1355] Step 7:

[1356] The emotion recognition engine determines the user's emotional state. The emotion recognition engine analyzes facial and voice data to determine the user's emotion. The input is the captured facial and voice data, and the output is the user's emotional state.

[1357] Step 8:

[1358] The server dynamically changes objects and effects in the virtual space based on the emotional data. For example, if the user makes a surprised expression, a specific effect is displayed. The input is the user's emotional data, and the output is the changed effects and objects in the virtual space.

[1359] Step 9:

[1360] Emotional data from multiple users is shared in real time. The server receives each user's emotional data and synchronizes it within the virtual space. This allows the emotions to be shared with other users, and corresponding effects are activated. The input is the emotional data of multiple users, and the output is the emotional data synchronized in real time and effects based on it.

[1361] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1362] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1363] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1364] [Fourth embodiment]

[1365] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1366] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1367] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1368] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1369] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1370] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1371] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1372] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1373] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1374] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1375] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1376] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1377] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1378] The present invention is a system that allows users to experience 2D data in 3D in a virtual space and share that experience among multiple users. This system mainly consists of the following steps.

[1379] 1. The user uploads 2D data

[1380] Users use their own devices to select and upload 2D data such as photos and images from their local disks, which the devices then receive and send to the server.

[1381] 2. The server receives the 2D data and sends it to the AI ​​model.

[1382] The server receives the uploaded 2D data, verifies the data format and file size, and then sends it to an AI model, which uses machine learning algorithms to analyze the 2D data and convert it into 3D data.

[1383] 3. Artificial intelligence models transform 2D data into 3D

[1384] The AI ​​model uses image analysis technology to identify the shape and depth of people and objects from 2D data. Based on this information, it generates a 3D mesh and applies textures to it, resulting in realistic 3D data.

[1385] 4. The server places the 3D data in the virtual space

[1386] The generated 3D data is placed in a virtual space by the server, and the space is ready for the user to view using a virtual reality device.

[1387] 5. The user experiences the virtual space using a virtual reality device.

[1388] Users wear virtual reality devices and launch a dedicated application to view and experience 3D data placed in a virtual space. Users can change their viewpoint and walk around the virtual space.

[1389] 6. Shared experiences among multiple users

[1390] Users can invite other users to the virtual space. Invited users can also use the same virtual reality device and application to experience the same virtual space. The server synchronizes the location and viewpoint data of multiple users in real time and displays each user's avatar, enabling a shared experience.

[1391] Specific examples

[1392] As a specific example, a case where photos of a family trip are used will be described.

[1393] User A: Uploads a group photo of a family trip from the device. The photo is a file called "familytrip.jpg." User A operates the device, selects the photo, and clicks the "Upload" button.

[1394] Server: Receives the upload request, saves "Family Trip.jpg", and sends it to the AI ​​model, which analyzes the photo, identifies the positions of family faces and the background scenery, and generates a 3D mesh.

[1395] Artificial intelligence model: Generates a 3D mesh and applies texture based on color information extracted from the original photo.

[1396] Server: Uploads the generated 3D data to the virtual space and links it to User A's profile.

[1397] User A: Puts on the VR goggles, launches the Metaverse application, and selects the "Family Trip" album. User A relives the family trip in a virtual space in a realistic way.

[1398] User A: Furthermore, User B invites Friend B to this virtual space. The server grants Friend B permission to view the space, and Friend B also puts on VR goggles and joins the virtual space. The two can relive memories of their family trip together and communicate in real time.

[1399] This system allows users to not only memorize memorable scenes as photographs, but also recreate them in three dimensions, enabling shared experiences in real time.

[1400] The processing flow will be explained below.

[1401] Step 1:

[1402] The user uses the terminal to select two-dimensional data (e.g., a photo file).

[1403] The user opens a browser or a dedicated application, accesses the photo upload screen, selects a file such as "Family Trip.jpg" from the local disk, and clicks the upload button.

[1404] Step 2:

[1405] The device sends the selected two-dimensional data to the server.

[1406] The device generates an HTTP POST request and sends the selected photo data and user ID to the server, along with metadata such as file name, file format, and file size.

[1407] Step 3:

[1408] The server receives and verifies the 2D data.

[1409] The server receives the HTTP POST request, saves the photo data in temporary storage, and performs validation checks such as checking for valid file formats and file sizes, and virus checks.

[1410] Step 4:

[1411] The server sends the photo data to an artificial intelligence model.

[1412] Once verified, the photo data is passed to the AI ​​model, which then sends the photo data and associated metadata to the AI ​​model.

[1413] Step 5:

[1414] An artificial intelligence model analyzes the two-dimensional data.

[1415] The AI ​​model identifies people and objects in the photo, analyzes their shape and depth, and uses deep learning algorithms to analyze each pixel in the photo and generate mesh data to recreate the three-dimensional structure.

[1416] Step 6:

[1417] An artificial intelligence model generates 3D meshes and textures.

[1418] A three-dimensional shape is created based on the mesh data. Color information is extracted from the original photo and texture is applied to the generated mesh. This results in a 3D model that looks realistic.

[1419] Step 7:

[1420] The server places the generated 3D data in a virtual space.

[1421] The completed 3D data is placed at a specific location in the virtual space. If multiple 3D objects are required, each object is placed in the appropriate location.

[1422] Step 8:

[1423] The server links the 3D data to the user's profile.

[1424] The generated 3D data ID is linked to the user ID and saved in a database, allowing the user to access the 3D data at any time.

[1425] Step 9:

[1426] The user puts on the virtual reality device and starts a dedicated application.

[1427] The user puts on the VR goggles and starts the Metaverse application. After the user is authenticated, a list of available albums is displayed within the application.

[1428] Step 10:

[1429] The user selects and views the generated 3D album.

[1430] Select "Family Trip" from the album list and view the 3D data placed in the virtual space. Users can change the viewpoint and walk around the virtual space.

[1431] Step 11:

[1432] A user sends a request to the server to invite others.

[1433] A user generates a request on their device to invite other users to the same virtual space and sends it to the server, which generates an invitation link or sharing code.

[1434] Step 12:

[1435] The server adds viewing permissions to other people's profiles.

[1436] The server receives the invitation request and adds the permission to view the three-dimensional data to the profile of the other specified user.

[1437] Step 13:

[1438] Other users access the virtual space using the invitation link.

[1439] Other users click on the invitation link and join the same virtual space using their virtual reality devices. The server synchronizes data in real time, and multiple avatars are displayed simultaneously.

[1440] Step 14:

[1441] The server synchronizes the location information and viewpoint data among multiple users.

[1442] The server receives each user's position and viewpoint data in real time and transmits that information to other users, allowing each user to see the avatars of other users in real time.

[1443] This series of steps allows users to re-experience a simple two-dimensional photo in a realistic three-dimensional space and share it with other users.

[1444] Example 1

[1445] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1446] Modern virtual reality technology is often experienced by users alone, and individual experiences tend to be fragmented. Furthermore, the process of converting 2D data into 3D and placing it in a virtual space is extremely complex and requires extensive specialized knowledge. As a result, it is difficult for ordinary users to create realistic 3D experiences from 2D data and share them with other users. Furthermore, it is difficult for multiple users to share an experience in real time in the same virtual space, limiting collaborative experiences.

[1447] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1448] In this invention, the server includes a means for users to upload 2D data from their devices, a means for receiving the 2D data and applying a generative AI model to convert it into 3D data, and a means for placing the converted 3D data in a virtual space. This allows general users to easily convert 2D data into 3D data, create realistic virtual experiences, and share them with other users. It also allows multiple users to experience 3D data in real time within the same virtual space, synchronizing each user's location information and viewpoint data. As a result, users can enjoy a collaborative virtual experience, enabling virtual reality technology to be enjoyed on a wider scale.

[1449] A "user" is a person who uses a terminal to upload two-dimensional data and experience virtual reality.

[1450] A "terminal" refers to an electronic device used by a user, such as a computer or smartphone.

[1451] "Two-dimensional data" refers to two-dimensional information such as photos and image data uploaded by users.

[1452] "Upload" refers to the act of a user sending data from a terminal to a server.

[1453] A "server" is a computer system that processes two-dimensional data received from users, converts it into three-dimensional data, and places it in a virtual space.

[1454] A "generative AI model" is a program that uses artificial intelligence technology to analyze two-dimensional data and convert it into three-dimensional data.

[1455] "3D data" refers to three-dimensional data converted from 2D data by a generative AI model.

[1456] "Virtual space" refers to a three-dimensional virtual space generated by computer graphics.

[1457] A "virtual reality device" refers to a head-mounted display or VR goggles that users use to experience a virtual space.

[1458] A "shared experience" is when multiple users simultaneously experience virtual reality in the same virtual space.

[1459] "Location information" refers to coordinate data within the user's virtual space.

[1460] "Viewpoint data" refers to data that indicates the visual direction and angle of the user within the virtual space.

[1461] An "invitation request" is a request made by a user to invite another user to a virtual space.

[1462] "Viewing authority" refers to the authority that allows an invited user to view data in a virtual space.

[1463] An "avatar" is a three-dimensional character that represents a user in a virtual space.

[1464] "Real-time synchronization" refers to the simultaneous updating of location information and viewpoint data for multiple users within a virtual space.

[1465] The present invention is a system that allows users to upload 2D data from their devices, convert it into 3D data, and share it in a virtual space. The steps for implementing this system are as follows:

[1466] First, the user uploads 2D data using a terminal. The terminal can be an electronic device such as a PC or smartphone. The user opens a file selection dialog, selects a photo or image data, such as "Family Trip.jpg," from the local disk, and clicks the upload button to send the data to the server. The technology used is an HTTP POST request.

[1467] Next, the server receives the 2D data. Using a web framework such as Flask or Django, the server stores the data appropriately and validates the data format and file size. The server then sends the 2D data to the artificial intelligence model for generative AI model application. The server uses a REST API or JSON format for this purpose.

[1468] The generative AI model uses machine learning libraries such as TensorFlow and PyTorch to analyze 2D data and convert it into 3D data. The AI ​​model analyzes the received 2D data to extract object contours and estimate depth. Based on the results of this analysis, it generates a 3D mesh and applies texture using color information extracted from the original image.

[1469] The server receives the generated 3D data and places it in a virtual space. The virtual space is built using a virtual space engine such as Unity or Unreal Engine. The server stores the generated 3D data in a database and places it appropriately in the virtual space.

[1470] Users wear virtual reality devices (such as Oculus Rift or HTC Vive) and view and experience 3D data in a virtual space. By launching a dedicated application, users can move freely within the virtual space and interact with it.

[1471] In addition, multiple users can share the experience of 3D data in the same virtual space. Users can invite other users to the virtual space, and the invited users also use virtual reality devices. The server synchronizes the location and viewpoint data of multiple users in real time and displays the avatars of other users.

[1472] Specific examples

[1473] As a concrete example, let's consider the case of a family trip photo. User A uploads a group photo of the family trip, "familytrip.jpg," from their device. The server receives the upload request, saves "familytrip.jpg," and sends it to the generative AI model. The generative AI model analyzes the photo, identifies the position of family members' faces and the background scenery, and generates a 3D mesh. The server uploads the generated 3D data to the virtual space and links it to User A's profile. User A puts on VR goggles, launches the Metaverse application, and selects the "Family Trip" album. Within the virtual space, they relive their family trip memories in a way that is close to reality. User A invites Friend B to this virtual space, and the server grants Friend B viewing permissions as well. Friend B also puts on VR goggles and joins the virtual space, and the two can experience their family trip memories together and communicate in real time.

[1474] Prompt Sentence Examples

[1475] What are the steps to upload photos from a family trip, turn them into 3D, experience them in a virtual space using a VR device, and share them with friends?

[1476] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1477] Step 1:

[1478] The user selects an image on the device

[1479] The user opens a file selection dialog using the terminal and selects an image to upload from the local disk (for example, "Family Trip.jpg"). The input is the 2D data (image file) selected by the user. The output is the path information of the selected image file.

[1480] Step 2:

[1481] The device sends the data to the server

[1482] When the user clicks the "Upload" button, the device sends the selected image data to the server as an HTTP POST request. Specifically, it uses the JavaScript fetch API to send the image data as form data. The input is the path information of the selected image file, and the output is the image data sent to the server.

[1483] Step 3:

[1484] The server receives the data

[1485] The server receives the image data sent from the device. It processes the HTTP POST request using a web framework such as Flask or Django. Specifically, the server temporarily stores the image data and verifies the format and size. The input is the image data in the HTTP request, and the output is the verified image data.

[1486] Step 4:

[1487] The server sends the data to the artificial intelligence model

[1488] The server sends the received image data to the generative AI model. Specifically, it uses a REST API to send the image data in JSON format. The input is the verified image data, and the output is an analysis request to the artificial intelligence model.

[1489] Step 5:

[1490] An artificial intelligence model analyzes the data

[1491] The generative AI model uses TensorFlow and PyTorch to perform image analysis. Specifically, it performs object recognition and depth estimation in the image to generate a 3D mesh. The input is 2D image data sent from the server, and the output is 3D mesh data.

[1492] Step 6:

[1493] Applying textures to 3D meshes

[1494] The AI ​​model applies texture to the generated 3D mesh using color information extracted from the original image. Specifically, it uses texture mapping technology. The input is 3D mesh data and texture information, and the output is the completed 3D data.

[1495] Step 7:

[1496] The server places the 3D data in the virtual space

[1497] The server receives the generated 3D data and places it in a virtual space created with Unity or Unreal Engine. Specifically, it stores the data in a database and transfers it to the virtual space engine. The input is the completed 3D data, and the output is 3D data placed in the virtual space.

[1498] Step 8:

[1499] Users experience a virtual space using a virtual reality device.

[1500] The user puts on the VR goggles and launches a dedicated application. Specific actions include visually viewing and manipulating objects in a virtual space. The input is 3D data placed in the virtual space, and the output is the visual information and interaction information of the virtual reality that the user experiences.

[1501] Step 9:

[1502] Shared experiences among multiple users

[1503] Users can invite other users into the virtual space, and the server synchronizes each user's location information and viewpoint data in real time. Specifically, the invited users also wear the same virtual reality device and use real-time communication technology. The input is the location information and viewpoint data of multiple users, and the output is a real-time shared virtual space experience.

[1504] Prompt Sentence Examples

[1505] What are the steps to upload photos from a family trip, turn them into 3D, experience them in a virtual space using a VR device, and share them with friends?

[1506] (Application example 1)

[1507] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1508] The present invention relates to a system that not only allows users to experience two-dimensional data in three dimensions within a virtual space, but also allows multiple users to share that experience in real time. In particular, the objective of the present invention is to provide a system that allows users to enjoy product experiences in a virtual store in three dimensions and share shopping experiences with other users.

[1509] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1510] In this invention, the server includes means for users to upload 2D data from their terminals, means for receiving the 2D data and applying an artificial intelligence model to convert it into 3D data, means for arranging the converted 3D data in a virtual space, means for users to experience the 3D data in the virtual space through a virtual reality device, means for multiple users to share the experience of the 3D data in the same virtual space, means for users to upload product images and view and experience products in a 3D manner in a virtual store, and means for inviting other users to the virtual store and sharing the shopping experience in real time, thereby enabling users to experience products in a 3D manner in the virtual store and share that experience with other users.

[1511] A "terminal" refers to an electronic device used by a user, such as a computer, smartphone, or tablet.

[1512] "Two-dimensional data" refers to digital data that contains two-dimensional information in terms of length and width, such as images and photographs.

[1513] "Upload" refers to the action of sending data from a user's terminal to a server.

[1514] "3D data" refers to three-dimensional digital data that has depth.

[1515] An "artificial intelligence model" is a data analysis technology that uses machine learning algorithms and has the ability to convert two-dimensional data into three-dimensional data.

[1516] A "virtual space" is a virtual environment that differs from reality and is constructed on a computer using digital technology.

[1517] A "virtual reality device" is a device that allows users to have an experience similar to the real world in a digital space, such as VR goggles or a head-mounted display.

[1518] "Product images" refer to photographs or digital images of products used in online shopping, etc.

[1519] A "virtual store" is a virtual shop that exists on the Internet where you can browse and purchase products.

[1520] "Viewing in 3D" refers to the act of using 3D data to display depth information, providing an experience that feels as if you are looking at the real thing.

[1521] A "shopping experience" is a series of online activities in which a user browses products and considers a purchase.

[1522] A "shared experience" refers to multiple users experiencing the same experience in real time in the same virtual space.

[1523] "Real-time" refers to processing and information exchange occurring immediately, without delay.

[1524] System Overview

[1525] The present invention relates to a system that allows users to experience two-dimensional data in a three-dimensional virtual space and share that experience among multiple users. In particular, the present invention provides a system that allows users to view products in a virtual store in three dimensions and share their shopping experience with other users.

[1526] Hardware and Software Used

[1527] Server: Use a high performance server, e.g. AWS EC2.

[1528] Virtual reality devices: Users use VR goggles or head-mounted displays. Example: Oculus Rift.

[1529] Devices: Smartphones, smart glasses, head-mounted displays, etc. Examples: Google Glass, HoloLens.

[1530] Machine learning libraries: TensorFlow, PyTorch.

[1531] 3D modeling software: Blender, Unity engine.

[1532] Details of data processing and calculation

[1533] 1. Data upload

[1534] The user uploads a product image using their device. For example, they send "New Product Bag.jpg" from their smartphone to the server. The API used for this is the REST API.

[1535] 2. Image analysis and 3D conversion

[1536] The server analyzes the received product images using machine learning algorithms. It uses a convolutional neural network (CNN)-based model to identify the shape and depth of the object in the image, then generates a 3D mesh and applies textures to it.

[1537] 3. Placement in virtual space

[1538] The server places the generated 3D data in the virtual store using the Unity engine, which optimally positions the 3D model in the virtual space.

[1539] 4. Virtual Experiences

[1540] Users wear VR goggles and experience a virtual store through the application. The server synchronizes the user's viewpoint data and location information in real time, allowing users to freely walk around the virtual store and experience the products in three dimensions.

[1541] 5. Shared Experiences

[1542] Users can invite other users to the virtual store. The server receives the invitation request and grants viewing privileges to other users. Location and viewpoint data are synchronized in real time between multiple users, allowing them to share their shopping experience.

[1543] Specific examples

[1544] User A uploads "New Product Bag.jpg" from their smartphone.

[1545] After receiving the image, the server converts it into a 3D bag using an artificial intelligence model and places it inside the virtual store.

[1546] User A puts on VR goggles and experiences the sensation of actually picking up a bag in a virtual store.

[1547] User A invites friend B to a virtual store and they shop together while viewing new products in real time.

[1548] Prompt Sentence Examples

[1549] Design an AI model that generates a 3D model from an image. The input is a product image file (e.g., "New Product Bag.jpg"), and the output is a 3D mesh and texture for display in a virtual space. The model must identify shape and depth from the image and generate a realistic 3D model. The 3D model must also be optimized for effective display in a virtual store.

[1550] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1551] Step 1:

[1552] The user uploads 2D data from their device. The input is a product image (e.g., "New Product Bag.jpg"), and the output is the image sent to the server. This data is sent to the server and received using a REST API.

[1553] Step 2:

[1554] The server validates the received 2D data. The input is the uploaded image file, and the output is the validation result to see if the data format and file size are correct. If the data is not in the correct format at this step, an error message is returned.

[1555] Step 3:

[1556] The server sends the data to an artificial intelligence model, which converts it into 3D data. The input is verified 2D data, and the output is 3D data with 3D mesh and texture. The model used here is a Convolutional Neural Network (CNN)-based analysis algorithm that identifies shape and depth to generate a 3D model.

[1557] Step 4:

[1558] The server places the generated 3D data in a virtual space. The input is 3D data, and the output is 3D data placed at specific coordinates in the virtual space. The Unity engine is used to render the 3D model in the virtual space.

[1559] Step 5:

[1560] Users wear virtual reality devices and experience 3D data in a virtual space. The input is the user's viewpoint data and position information, and the output is an interactive experience in the virtual space. The server synchronizes this data in real time, allowing users to move freely within the virtual space.

[1561] Step 6:

[1562] Users can invite other users to a virtual space to share experiences. The input is an invitation request and the other user's ID, and the output is the granting of viewing permissions to the other users. Upon receiving the invitation, the server synchronizes the location and viewpoint data of multiple users in real time, providing a shared experience in the virtual space.

[1563] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1564] This invention is a system that allows users to experience 2D data in 3D in a virtual space and share that experience with multiple users, and also adds technology that recognizes the user's emotions and dynamically changes the objects and effects in the virtual space according to the emotions. This system mainly consists of the following steps.

[1565] 1. The user uploads 2D data

[1566] Users use their own devices to select and upload 2D data such as photos and images from their local disks, which the devices then receive and send to the server.

[1567] 2. The server receives the 2D data and sends it to the AI ​​model.

[1568] The server receives the uploaded 2D data, verifies the data format and file size, and then sends it to an AI model, which uses machine learning algorithms to analyze the 2D data and convert it into 3D data.

[1569] 3. Artificial intelligence models transform 2D data into 3D

[1570] The AI ​​model uses image analysis technology to identify the shape and depth of people and objects from 2D data. Based on this information, it generates a 3D mesh and applies textures to it, resulting in realistic 3D data.

[1571] 4. The server places the 3D data in the virtual space

[1572] The generated 3D data is placed in a virtual space by the server, and the space is ready for the user to view using a virtual reality device.

[1573] 5. The user experiences the virtual space using a virtual reality device.

[1574] Users wear virtual reality devices and launch a dedicated application to view and experience 3D data placed in a virtual space. Users can change their viewpoint and walk around the virtual space.

[1575] 6. Shared experiences among multiple users

[1576] Users can invite other users to the virtual space. Invited users can also use the same virtual reality device and application to experience the same virtual space. The server synchronizes the location and viewpoint data of multiple users in real time and displays each user's avatar, enabling a shared experience.

[1577] 7. Introducing an emotion recognition engine

[1578] The user's facial expression is captured using a camera and microphone, and the captured facial expressions and vocal tone are sent to an emotion recognition engine, which analyzes the data and determines the user's emotional state in real time.

[1579] 8. Changes in virtual space based on emotional data

[1580] Based on the recognized emotion data, objects and effects in the virtual space are dynamically changed. For example, if the user makes a surprised expression, all the objects in the virtual space will move in unison, or special effects will be displayed.

[1581] 9. Emotion Sharing among Multiple Users

[1582] When multiple users are in the same virtual space, they share their emotional data in real time. The server receives this emotional data and reflects it on each user's avatar and environment, making it easier for other users to see their emotional state.

[1583] Specific examples

[1584] As a specific example, a case where photos of a family trip are used will be described.

[1585] User A: Uploads a group photo of a family trip from the device. The photo is a file called "familytrip.jpg." User A operates the device, selects the photo, and clicks the "Upload" button.

[1586] Server: Receives the upload request, saves "Family Trip.jpg", and sends it to the AI ​​model, which analyzes the photo, identifies the positions of family faces and the background scenery, and generates a 3D mesh.

[1587] Artificial intelligence model: Generates a 3D mesh and applies texture based on color information extracted from the original photo.

[1588] Server: Uploads the generated 3D data to the virtual space and links it to User A's profile.

[1589] User A: Puts on the VR goggles, launches the Metaverse application, and selects the "Family Trip" album. User A relives the family trip in a virtual space in a realistic way.

[1590] Emotion recognition engine: Monitors user A's facial expressions and voice and analyzes their emotional state in real time. For example, if user A smiles, a character in the virtual space may greet them.

[1591] User A: Then, he invites Friend B to join the virtual space. The server grants Friend B permission to view the content, and Friend B also puts on VR goggles and joins the virtual space. The two can relive memories of their family trip together and share their emotions in real time. For example, if User A makes a surprised expression, that facial expression data is transmitted to Friend B, and a surprise effect may be displayed in the virtual space.

[1592] This system allows users to not only memorize memorable scenes as two-dimensional photographs, but also recreate them in three dimensions, allowing them to be experienced and shared in a virtual space that changes dynamically according to the user's emotions.

[1593] The processing flow will be explained below.

[1594] Step 1:

[1595] The user uses the terminal to select two-dimensional data (e.g., a photo file).

[1596] The user opens a browser or a dedicated application, accesses the photo upload screen, selects a file such as "Family Trip.jpg" from the local disk, and clicks the upload button.

[1597] Step 2:

[1598] The device sends the selected two-dimensional data to the server.

[1599] The device generates an HTTP POST request and sends the selected photo data and user ID to the server, along with metadata such as file name, file format, and file size.

[1600] Step 3:

[1601] The server receives and verifies the 2D data.

[1602] The server receives the HTTP POST request, saves the photo data in temporary storage, and performs validation checks such as checking for valid file formats and file sizes, and virus checks.

[1603] Step 4:

[1604] The server sends the photo data to an artificial intelligence model.

[1605] Once verified, the photo data is passed to the AI ​​model, which then sends the photo data and associated metadata to the AI ​​model.

[1606] Step 5:

[1607] An artificial intelligence model analyzes the two-dimensional data.

[1608] The AI ​​model identifies people and objects in the photo, analyzes their shape and depth, and uses deep learning algorithms to analyze each pixel in the photo and generate mesh data to recreate the three-dimensional structure.

[1609] Step 6:

[1610] An artificial intelligence model generates 3D meshes and textures.

[1611] A three-dimensional shape is created based on the mesh data. Color information is extracted from the original photo and texture is applied to the generated mesh. This results in a 3D model that looks realistic.

[1612] Step 7:

[1613] The server places the generated 3D data in a virtual space.

[1614] The completed 3D data is placed at a specific location in the virtual space. If multiple 3D objects are required, each object is placed in the appropriate location.

[1615] Step 8:

[1616] The server links the 3D data to the user's profile.

[1617] The generated 3D data ID is linked to the user ID and saved in a database, allowing the user to access the 3D data at any time.

[1618] Step 9:

[1619] The user puts on the virtual reality device and starts a dedicated application.

[1620] The user puts on the VR goggles and starts the Metaverse application. After the user is authenticated, a list of available albums is displayed within the application.

[1621] Step 10:

[1622] The user selects and views the generated 3D album.

[1623] Select "Family Trip" from the album list and view the 3D data placed in the virtual space. Users can change the viewpoint and walk around the virtual space.

[1624] Step 11:

[1625] A user sends a request to the server to invite others.

[1626] A user generates a request on their device to invite other users to the same virtual space and sends it to the server, which generates an invitation link or sharing code.

[1627] Step 12:

[1628] The server adds viewing permissions to other people's profiles.

[1629] The server receives the invitation request and adds the permission to view the three-dimensional data to the profile of the other specified user.

[1630] Step 13:

[1631] Other users access the virtual space using the invitation link.

[1632] Other users click on the invitation link and join the same virtual space using their virtual reality devices. The server synchronizes data in real time, and multiple avatars are displayed simultaneously.

[1633] Step 14:

[1634] The server synchronizes the location information and viewpoint data among multiple users.

[1635] The server receives each user's position and viewpoint data in real time and transmits that information to other users, allowing each user to see the avatars of other users in real time.

[1636] Step 15:

[1637] Users share experiences with other users in a virtual space.

[1638] Users can communicate with other users in a virtual space while viewing and experiencing the same 3D data.

[1639] Step 16:

[1640] The emotion recognition engine collects the user's facial expressions and voice data.

[1641] The emotion recognition engine uses the camera and microphone installed in the VR goggles to capture the user's facial expressions and voice tone, thereby grasping the user's real-time emotional state.

[1642] Step 17:

[1643] An emotion recognition engine analyzes the user's emotions.

[1644] The emotion recognition engine analyzes the collected data and determines what emotions the user is feeling, such as smiling, surprised, or sad.

[1645] Step 18:

[1646] The server reflects the recognized emotions in the virtual space.

[1647] When the user's emotion is recognized, the server applies that data to the virtual space, causing objects and effects in the virtual space to change dynamically. For example, if the user smiles, an effect that brightens the scenery is applied.

[1648] Step 19:

[1649] The server shares emotion data among multiple users.

[1650] When multiple users exist in a virtual space, the server also synchronizes each user's emotional data in real time, allowing each user to see the emotional state of the other users.

[1651] Step 20:

[1652] Other users experience the shared emotion data.

[1653] Other users in the virtual space can experience the emotional changes in real time and adjust their own actions and reactions accordingly.

[1654] This series of steps allows users to re-experience a simple two-dimensional photograph in a realistic three-dimensional space, and share the experience with other users in a virtual space that changes dynamically with changes in emotions.

[1655] Example 2

[1656] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1657] While existing technologies exist for systems that allow users to experience 2D real-world data (such as photos and images) in 3D virtual space, they lack the ability to recognize users' emotions in real time and dynamically change objects and effects in the virtual space accordingly, as well as the ability to share emotions in real time among multiple users. For this reason, there is a demand for systems that can make user experiences richer and more interactive.

[1658] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for a user to upload 2D data from a terminal, means for receiving the 2D data and verifying the data format and file size before transmitting it to an artificial intelligence model, means for analyzing the 2D data using the artificial intelligence model and converting it into 3D data, means for arranging the converted 3D data in a virtual space, means for a user to access the 3D data in the virtual space via a virtual reality device to view and experience it, means for multiple users to share and experience the 3D data in the same virtual space, means for collecting emotion data using a user's facial expression recognition camera or microphone and transmitting it to an emotion recognition engine, means for dynamically changing objects and effects in the virtual space based on the recognized emotion data, and means for sharing and reflecting emotion data with other users in real time. This enables dynamic changes in the virtual space according to the user's emotions and emotion sharing among multiple users.

[1659] A "terminal" is an electronic device used by a user, and includes forms such as a personal computer, a smartphone, and a tablet.

[1660] "Two-dimensional data" refers to data that exists on a plane, such as photographs, images, and graphics.

[1661] A "server" is a computer system that communicates with multiple terminals via a network and stores, processes, sends and receives data.

[1662] "Data format" refers to the method or structure in which digital data is stored, including image formats such as JPEG and PNG.

[1663] "File size" is an indicator of the size of data and is expressed in units such as kilobytes (KB) or megabytes (MB).

[1664] An "artificial intelligence model" refers to a software system that uses machine learning algorithms or deep learning to perform specific tasks.

[1665] "Image analysis" is the process of extracting and identifying features and patterns in two-dimensional data, and primarily uses deep learning technology.

[1666] "Three-dimensional data" is data that has depth information, including the shape and position of a three-dimensional object.

[1667] A "3D mesh" is a data structure that represents the surface of a 3D object and is composed of a large number of polygons (mainly triangles).

[1668] "Texture" refers to image or color information applied to a 3D mesh to make the object appear more realistic.

[1669] A "virtual space" is a computer-generated three-dimensional environment that users can visually experience.

[1670] "Virtual reality devices" refer to devices for experiencing virtual spaces, such as VR goggles and head-mounted displays (HMDs).

[1671] An "expression recognition camera" is a camera device that detects a user's facial expressions and captures them as data.

[1672] A "microphone" is a device for collecting sound and plays a role in transmitting the user's voice to the emotion recognition engine.

[1673] An "emotion recognition engine" is a software system that analyzes data such as facial expressions and tone of voice to recognize a user's emotional state.

[1674] "Emotional data" is data that represents the user's emotional state and is collected and analyzed in real time.

[1675] An "object" refers to any three-dimensional model or entity that exists in a virtual space.

[1676] "Effects" are visual effects that are displayed in response to specific events or actions within the virtual space.

[1677] "Location information" refers to the coordinate information of a user or avatar within a virtual space.

[1678] "Viewpoint data" is data that represents the user's line of sight and the camera direction.

[1679] This invention is a system that allows users to experience two-dimensional data in three dimensions within a virtual space and share that experience among multiple users, and also adds technology that recognizes the user's emotions and dynamically changes the objects and effects within the virtual space in response to those emotions.

[1680] An embodiment of this system includes the following process.

[1681] 1. User uploads 2D data

[1682] Users use their own devices (e.g., personal computers, smartphones, tablets) to select and upload 2D data such as photos and images from their local disks. Specifically, users click the file selection button and select the image file they want to upload. After making their selection, they press the "Upload" button, which causes the device to send a request to the server.

[1683] 2. Receipt and verification of data by the server

[1684] The server receives the upload request sent from the device. Then, it verifies the file format and size of the received 2D data. For example, if the format is invalid or the file size is too large, it returns an error message to the user.

[1685] 3. Sending data from the server to the AI ​​model

[1686] The server then sends the verified 2D data to the AI ​​model, preprocessing the data (e.g., resizing the image or converting the format). After sending, the server records the progress and may notify the user in real time.

[1687] 4. Analyzing and transforming data using artificial intelligence models

[1688] The AI ​​model uses image analysis techniques to identify the shape and depth of people and objects from 2D data. Specifically, it uses deep learning algorithms (e.g., convolutional neural networks) to extract feature points within the image. It then generates a 3D mesh based on these feature points and applies a texture based on color information extracted from the original photo.

[1689] 5. Placement of 3D data in virtual space by the server

[1690] The server places the 3D data received from the AI ​​model in the virtual space. Specifically, it places the data according to the virtual space's coordinate system and sets appropriate position information, allowing users to smoothly browse the virtual space using a virtual reality device.

[1691] 6. User Experience of Virtual Space

[1692] The user puts on the virtual reality device (e.g., VR goggles, head-mounted display) and launches a dedicated application. Specifically, they click the "Start Experience" button from the menu within the application and move into the virtual space. The user can change their viewpoint and walk around the virtual space using a controller.

[1693] 7. Shared experiences among multiple users

[1694] Users can invite other users to the virtual space by clicking the "Invite" button in the dedicated application, entering their friend's user ID, and sending an invitation link. Invited users can also use virtual reality devices and the application to experience the same virtual space. The server synchronizes the location and viewpoint data of multiple users in real time, and displays each user's avatar, enabling a shared experience.

[1695] 8. How the Emotion Recognition Engine Works

[1696] The server sends data collected from the facial recognition camera and microphone installed on the user's device to the emotion recognition engine. For example, the device uses the front camera to detect the user's facial expressions. In the case of audio data, the audio recorded by the microphone is sent. The emotion recognition engine analyzes this data and determines the user's emotional state in real time.

[1697] 9. Changes in virtual space based on emotional data

[1698] The server dynamically changes the objects and effects in the virtual space based on the emotional data obtained from the emotion recognition engine. If the user makes a surprised expression, the objects in the virtual space will move all at once, or special effects (e.g., fireworks or light effects) will be displayed.

[1699] 10. Emotion Sharing among Multiple Users

[1700] The server receives emotional data from multiple users in real time and shares it within the virtual space. User A's emotional data is also displayed to User B, allowing them to mutually confirm their emotional state. For example, if User A smiles, User B's avatar will also smile.

[1701] Reproducing specific examples

[1702] As a concrete example, we will explain the case where a photo of a family trip is used. User A uploads a group photo of the family trip, "familytrip.jpg", from his / her terminal. The detailed procedure is explained below.

[1703] Upload steps:

[1704] User A operates the terminal to select the family trip photo "familytrip.jpg" and clicks the "upload" button.

[1705] The server receives the upload request and sends it to the artificial intelligence model.

[1706] Data analysis and transformation:

[1707] An artificial intelligence model analyzes the photo, identifies the position of family members' faces and the background scenery, and generates a 3D mesh.

[1708] A texture is applied to the generated 3D mesh based on color information extracted from the original photo.

[1709] Placement in virtual space:

[1710] The server uploads the generated 3D data to the virtual space and links it to User A's profile.

[1711] Virtual space experience:

[1712] User A puts on the VR goggles, launches the Metaverse application, and selects the "Family Trip" album. In the virtual space, he or she relives memories of the family trip in a realistic way.

[1713] Emotion Recognition and Feedback:

[1714] The emotion recognition engine monitors the facial expressions and voice of user A and analyzes their emotional state in real time. For example, if user A smiles, a character in the virtual space may greet them.

[1715] Adding a shared experience:

[1716] User A invites Friend B to this virtual space, and the server grants Friend B viewing permission to join the virtual space. The two can relive memories of a family trip and share their emotions in real time. If User A makes a surprised expression, that facial expression data is transmitted to Friend B, and a surprise effect may be displayed in the virtual space.

[1717] Using this system, it is possible to realize an experience that goes beyond simple two-dimensional photographs, and instead involves a three-dimensional virtual space that changes dynamically according to the user's emotions.

[1718] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1719] Step 1: User uploads 2D data

[1720] Input: 2D data selected by the user (e.g., photo "Family Trip.jpg")

[1721] Output: Upload request from device to server

[1722] Specific operation: The user operates the device, clicks the file selection button, and selects 2D data such as a photo or image. After making a selection, the user presses the "Upload" button, which sends the selected data to the server. The device uploads this data as an HTTP request to the server in the specified format.

[1723] Step 2: Server receives and verifies data

[1724] Input: Upload request sent from the device (2D data)

[1725] Output: Validated 2D data or error message

[1726] Specific operation: After the server receives the upload request, it verifies the data format (e.g., JPEG, PNG) and file size (e.g., less than 2MB). If the format is invalid or the file size is too large, it returns an error message to the user. If the verification is successful, the received data proceeds to the next step.

[1727] Step 3: Sending data from the server to the AI ​​model

[1728] Input: Verified 2D data

[1729] Output: A request to send data to the AI ​​model

[1730] How it works: The server sends the verified 2D data to the AI ​​model. If necessary, it preprocesses the data (e.g., resizes or converts the image). The preprocessed data is sent to the AI ​​model and progress is recorded. The user may be notified of progress in real time.

[1731] Step 4: Analyze and transform the data with artificial intelligence models

[1732] Input: 2D data sent from the server

[1733] Output: 3D mesh, 3D data with texture information

[1734] How it works: The AI ​​model uses image analysis techniques to identify the shape and depth of people and objects from 2D data. It then uses deep learning algorithms (e.g., convolutional neural networks) to extract feature points within the image. This generates a 3D mesh and applies texture based on color information extracted from the original photo.

[1735] Step 5: The server places the 3D data in the virtual space.

[1736] Input: 3D data sent from an artificial intelligence model

[1737] Output: 3D data placed in virtual space

[1738] Specific operation: The server places the 3D data received from the AI ​​model in the virtual space. The server aligns the data with the virtual space's coordinate system and sets appropriate position information. This allows the user to smoothly browse the virtual space using a virtual reality device.

[1739] Step 6: User experience of the virtual space

[1740] Input: 3D data placed in virtual space

[1741] Output: User experience data (viewpoint movement, action data)

[1742] Specific operation: The user puts on the virtual reality device (e.g., VR goggles) and launches the dedicated application. They click the "Start Experience" button from the application's menu and move into the virtual space. The user walks around the virtual space, changes their viewpoint, and operates the controller. This experience data is recorded in real time.

[1743] Step 7: Shared experiences with multiple users

[1744] Input: User invitation request, location and viewpoint data of other users

[1745] Output: Synchronized experience data for multiple users

[1746] How it works: A user clicks the "Invite" button in the dedicated application, enters their friend's user ID, and sends an invitation link. The invited user also uses the virtual reality device and application to experience the same virtual space. The server synchronizes the location and viewpoint data of multiple users in real time, displays each user's avatar, and enables a shared experience.

[1747] Step 8: Emotion Recognition Engine in Action

[1748] Input: User's facial expression data and voice data

[1749] Output: Parsed emotion data

[1750] Specific operation: The server sends data collected from the facial recognition camera and microphone installed on the user's device to the emotion recognition engine. For example, the front camera is used to detect the user's facial expressions, and voice data is recorded by the microphone. The emotion recognition engine analyzes this data and determines the user's emotional state in real time.

[1751] Step 9: Change the virtual space based on emotion data

[1752] Input: Emotion data sent from the emotion recognition engine

[1753] Output: Dynamically changing virtual space objects and effects

[1754] Specific operation: The server dynamically changes the objects and effects in the virtual space based on the emotional data obtained from the emotion recognition engine. For example, if the user makes a surprised expression, the objects in the virtual space will all move in unison and special effects will be displayed.

[1755] Step 10: Emotion sharing among multiple users

[1756] Input: Emotion data from multiple users

[1757] Output: Changes in the virtual space based on shared emotional data

[1758] Specific operation: The server receives emotional data from multiple users in real time and shares it in the virtual space. For example, the emotional data of user A can be displayed to user B, allowing them to understand each other's emotional states. When user A smiles, user B's avatar also smiles. In this way, emotions can be shared among multiple users.

[1759] (Application example 2)

[1760] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1761] The present invention relates to a system that uses two-dimensional data to provide a three-dimensional experience and allows users to share that experience. However, conventional systems have the problem of being unable to dynamically change the environment or objects in the virtual space in response to the user's emotions, resulting in a lack of immersion and emotional sharing between users. Furthermore, when multiple users share an experience in a virtual reality space, the lack of real-time sharing of emotional data and changes to the environment based on those emotions limits the experience.

[1762] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for a user to upload 2D data from a terminal, means for receiving the 2D data and applying an artificial intelligence model to convert the 2D data into 3D data, means for arranging the converted 3D data in a virtual space, means for a user to experience the 3D data in the virtual space via a virtual reality device, means for multiple users to share and experience the 3D data in the same virtual space, means for recognizing a user's emotion and dynamically changing objects and effects in the virtual space based on the emotion, and means for sharing emotion data of multiple users in real time and dynamically changing effects in the virtual space based on the data. As a result, the environment and objects in the virtual space dynamically change according to the user's emotion, enabling emotions to be shared among multiple users.

[1763] "Two-dimensional data" is digital data that is represented on a plane, such as photographs, images, and videos.

[1764] "3D data" is data that has depth and a sense of three-dimensionality, and is digital data that includes meshes and textures.

[1765] An "artificial intelligence model" is an algorithm that uses machine learning and deep learning, and is a computational tool for analyzing two-dimensional data and converting it into three-dimensional data.

[1766] A "virtual space" is a computer-generated, three-dimensional digital environment that a user can experience through a virtual reality device.

[1767] A "virtual reality device" is a device that allows users to experience data in a virtual space, such as a VR headset or HMD (Head-Mounted Display).

[1768] An "emotion recognition engine" is software or hardware that analyzes a user's facial expressions and voice to determine their emotional state.

[1769] "Effects" are visual or auditory effects displayed within the virtual space, which are dynamically generated according to the user's emotional state.

[1770] A "3D mesh" is a polygon structure formed by vertices, edges, and faces that make up a three-dimensional shape, and is a data structure that provides an actual sense of three-dimensionality.

[1771] A "texture" is image data that is applied to a 3D mesh to realistically represent the texture and color of an object's surface.

[1772] An "avatar" is a digital character that represents a user in a virtual space.

[1773] "Emotion data" is data that indicates the emotional state of the user analyzed by an emotion recognition engine.

[1774] "Real-time synchronization" means that the location information, viewpoint data, emotional data, etc. of multiple users in a virtual space are updated simultaneously, allowing all users to share the latest information.

[1775] The present invention is a system that allows users to experience 2D data in 3D within a virtual space, share that experience among multiple users, and dynamically change objects and effects within the virtual space based on the user's emotions. The realization of this system involves the following means and processes.

[1776] First, users use their devices to upload two-dimensional data, which refers to digital data represented on a flat surface, such as photographs, images, and videos, which are selected and uploaded through the device's user interface.

[1777] The server then receives the uploaded 2D data, verifies the data, and then converts it into 3D data using a generative AI model. A generative AI model is an algorithm designed using machine learning or deep learning that analyzes 2D image data, estimates 3D shapes, and generates 3D meshes and textures based on the 3D shapes.

[1778] The server then places the generated 3D data in a virtual space, which is a computer-generated 3D digital environment that users can experience through a virtual reality device. Users can wear the device and view and experience the 3D data in the virtual space.

[1779] Another feature of the present invention is an emotion recognition engine that recognizes a user's emotions in real time. An emotion recognition engine is software or hardware for analyzing a user's emotional state from their facial expressions and voice. The server collects the user's facial expressions and voice and sends them to the emotion recognition engine. The emotion recognition engine analyzes this data and determines the user's emotions.

[1780] Based on the emotional data, the server dynamically changes the objects and effects in the virtual space. For example, if the user expresses surprise, objects in the virtual space may suddenly move or visual effects may be displayed. This makes the user's experience more intuitive and immersive.

[1781] Furthermore, the system includes a means for sharing emotional data among multiple users in real time. When multiple users are experiencing the same virtual space, they can share their emotional data and dynamically change the effects in the virtual space based on that data. This allows multiple users to share a common experience and feel a sense of emotional unity.

[1782] A specific use case is a scenario where a user uploads a video of a family trip, converts it into 3D, and relives it using a virtual reality device. When the emotion recognition engine recognizes the user's smile, effects such as fireworks appear in the virtual space. Friends can also be invited to the same virtual space to enjoy the experience together.

[1783] Example prompt sentence:

[1784] "Upload a video of your family trip"

[1785] "Displays various effects when the face is surprised"

[1786] "Invite friends to join you in VR"

[1787] This allows users to enjoy a richer, more shareable virtual reality experience.

[1788] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1789] Step 1:

[1790] The user uploads 2D data using the terminal. Specifically, the user selects a 2D data file such as a photo or video and clicks the upload button, and the terminal sends the selected data to the server. The input is the 2D data file, and the output is the data sent to the server.

[1791] Step 2:

[1792] The server receives the 2D data. The server verifies the format and file size of the received data to ensure it is valid. The input is the 2D data file sent from the terminal, and the output is the verified data.

[1793] Step 3:

[1794] The server sends the verified 2D data to the generative AI model, which uses machine learning algorithms to analyze the 2D data and convert it into 3D data. The input is a 2D data file, and the output is 3D data.

[1795] Step 4:

[1796] The server places the generated 3D data in a virtual space. The server then places the 3D mesh and texture in the appropriate position within the virtual space, allowing the user to experience it using a virtual reality device. The input is the 3D data output by the generative AI model, and the output is 3D data placed within the virtual space.

[1797] Step 5:

[1798] The user wears a virtual reality device and experiences 3D data in a virtual space. The user puts on a VR headset and launches a dedicated application to enter the virtual space. The input is 3D data placed in the virtual space, and the output is the user's experience.

[1799] Step 6:

[1800] The server sends the user's facial expressions and voice to the emotion recognition engine. The camera and microphone installed in the virtual reality device capture the user's facial expressions and voice in real time and send them to the server. The server then sends them to the emotion recognition engine. The input is the user's facial expression and voice data, and the output is the input data for the emotion recognition engine.

[1801] Step 7:

[1802] The emotion recognition engine determines the user's emotional state. The emotion recognition engine analyzes facial and voice data to determine the user's emotion. The input is the captured facial and voice data, and the output is the user's emotional state.

[1803] Step 8:

[1804] The server dynamically changes objects and effects in the virtual space based on the emotional data. For example, if the user makes a surprised expression, a specific effect is displayed. The input is the user's emotional data, and the output is the changed effects and objects in the virtual space.

[1805] Step 9:

[1806] Emotional data from multiple users is shared in real time. The server receives each user's emotional data and synchronizes it within the virtual space. This allows the emotions to be shared with other users, and corresponding effects are activated. The input is the emotional data of multiple users, and the output is the emotional data synchronized in real time and effects based on it.

[1807] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1808] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1809] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1810] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1811] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1812] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1813] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1814] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ide...

Claims

1. A means for a user to upload two-dimensional data from a terminal; means for receiving the two-dimensional data and applying an artificial intelligence model to convert the data into three-dimensional data; A means for placing the converted three-dimensional data in a virtual space; A means for a user to experience three-dimensional data in a virtual space through a virtual reality device; A means for multiple users to share and experience 3D data in the same virtual space; A system including:

2. means for analyzing the uploaded two-dimensional data and identifying the shape and depth of people and objects; a means for generating a 3D mesh based on the identified shape and depth; means for applying a texture to the generated 3D mesh and mapping color information extracted from the original 2D data; The system of claim 1 , comprising:

3. a means for synchronizing position information and viewpoint data of multiple users in real time and displaying avatars of other users; means for receiving an invitation request from a user and granting viewing privileges to other users; The system of claim 1 , comprising:

Images