System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving photographic and video data; means for using generative artificial intelligence to transform the photographic and video data into three-dimensional data; means for generating a virtual reality space from the three-dimensional data; and means for providing an interface for a user to access and experience the virtual reality space.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional photographs and videos are flat, making it difficult to experience memories more deeply. Furthermore, there are limited means for recreating shared memories with family and friends. This makes it difficult to connect with family members who live far away, relive memories with deceased relatives, and preserve a child's growth in a realistic way. The present invention aims to solve these problems and provide a way to relive memories more realistically and easily. [Means for solving the problem]

[0005] The present invention provides a system including means for receiving photo data and video data, means for using artificial intelligence to convert the photo data and video data into three-dimensional data, means for generating a virtual reality space from the three-dimensional data, and means for providing an interface for users to access and experience the virtual reality space. This system allows users to easily convert conventional two-dimensional photos and videos into three-dimensional data, and further generate and experience a virtual reality space based on that data. This allows users to realistically relive shared memories with family and friends who live far away.

[0006] "Photo data" refers to data that expresses captured image information in a digital format.

[0007] "Moving image data" is data that represents a series of image frames in a digital format and can be played back as moving images.

[0008] "Three-dimensional data" refers to data that includes shape and position information of an object or space and can be expressed in three-dimensional space.

[0009] "Generative AI" is an AI technology that has the ability to generate new information and data based on given data.

[0010] A "virtual reality space" is a virtual space that uses digital technology to simulate environments and spaces that are different from reality, allowing users to immerse themselves in them.

[0011] An "interface" is the operating means and display mechanism that allows a user to interact with a system.

[0012] "Metadata" is additional information about data, and is data that primarily includes information that describes the characteristics and context of the data.

[0013] A "three-dimensional model" is digital data or a representation thereof that represents the shape and structure of an object in three-dimensional space. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] This invention provides a system that converts photo data and video data into three-dimensional data and generates a virtual reality space based on that data. The system of the present invention is composed of three main elements: a user, a terminal, and a server.

[0036] First, the user selects and uploads family photos and videos from their device. The user can optionally enter metadata such as the shooting location and date and time. For example, the user can select photos and videos from the "Family Trip" folder and enter "Hawaii" as the shooting location.

[0037] Next, the device sends the file selected by the user and the metadata entered to the server. The server receives and stores the file data sent by the user. Based on the received file data, the server uses generative AI to convert the photo and video data into 3D data. For example, generative AI analyzes 2D photos and generates a 3D model of a family. It also does the same for video data, generating moving 3D data.

[0038] The server also generates a 3D model of the shooting location based on the metadata as needed. For example, if "Hawaii" is entered as metadata, a 3D model recreating the scenery and tourist spots of Hawaii is generated. A virtual reality space is then constructed based on this 3D model.

[0039] The server then saves the generated 3D data and virtual reality space in a user-accessible state and generates a preview link. The user can access the generated 3D data and virtual reality space and view it on their own device. For example, the user can click the preview link to view the 3D image or video in a browser.

[0040] Furthermore, users can use VR goggles to immerse themselves in the virtual reality space and relive their past experiences. They can also invite other family members and friends to the virtual reality space and relive shared memories in an immersive way. For example, a user can send an invitation link to their family members, allowing them to explore a virtual Hawaiian space together and share memories.

[0041] This system allows users to easily turn conventional flat photos and videos into three dimensions, allowing them to relive memories in a more immersive way. It also has the advantage of deepening interactions with family and friends, allowing them to share realistic experiences with family members who live far away.

[0042] For example, if a user selects photos and videos from a "Family Trip" folder and enters metadata about Hawaii, the system works as follows: The user first uploads the files from their device to the server, which receives and stores the data. The generative AI then creates a 3D model of Hawaii and constructs a virtual reality space. The user can then access the 3D data and virtual reality space and share the immersive experience with family and friends through VR goggles.

[0043] As described above, the present invention provides users with a means for reliving memories in a more realistic and convenient manner.

[0044] The processing flow will be explained below.

[0045] Step 1:

[0046] The user selects photos and videos from their device and presses the upload button. If necessary, the user can enter metadata such as the shooting location and date and time.

[0047] Step 2:

[0048] The terminal sends the selected file data and input metadata to the server, which receives the request and stores the file data and metadata.

[0049] Step 3:

[0050] The server sends the stored photo and video data to the generation AI, which analyzes the 2D data and generates a corresponding 3D model. Through this process, the generation AI outputs the three-dimensional data.

[0051] Step 4:

[0052] The server analyzes the metadata and generates a 3D model of the location where the photo was taken, if necessary. For example, if the metadata includes "Hawaii," it generates a 3D model of Hawaiian scenery and tourist attractions.

[0053] Step 5:

[0054] The server combines the generated 3D data with the virtual reality space and stores it as data for each user. It also generates a preview link so that the user can access this data.

[0055] Step 6:

[0056] Users click the preview link to view the 3D image or video. The device downloads the 3D data from the server and displays it in the browser.

[0057] Step 7:

[0058] Users put on VR goggles and log in to the virtual reality space, where they can enjoy an immersive experience with their family and friends.

[0059] Step 8:

[0060] The user invites other family members and friends to the virtual reality space, and the device sends an invitation link to the family and friends, allowing everyone to access the virtual reality space.

[0061] The above is the specific flow of the program's processing, which allows the user to re-experience memories in a three-dimensional and realistic way.

[0062] Example 1

[0063] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0064] Conventional photos and videos are two-dimensional, and the visual information they provide is limited, making it difficult for users to feel a sense of presence or a deep experience. Furthermore, the means for sharing those experiences with family and friends who live far away are also limited. This makes it difficult to recreate memories in a more realistic and immersive way, and to deepen shared experiences.

[0065] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0066] In this invention, the server includes means for receiving photo data and video data, means for using artificial intelligence to convert the photo data and video data into three-dimensional data, means for generating a virtual reality space based on the three-dimensional data and metadata, and means for providing an interface for users to access and experience the virtual reality space. This allows users to convert two-dimensional photos and videos into three-dimensional space, re-create memories in an immersive way, and share realistic experiences with family and friends who live far away.

[0067] "Photo data and video data" refers to digital files of still images and videos taken by users.

[0068] "Means for receiving" refers to the structure or method for transferring and storing digital files from a user's terminal to a server.

[0069] "Means using generative artificial intelligence" refers to a structure or method for converting two-dimensional photographs or videos into three-dimensional data using artificial intelligence techniques.

[0070] "Three-dimensional data" refers to a three-dimensional digital object created based on two-dimensional information from photographs and videos.

[0071] "Means for generating a virtual reality space" refers to a structure or method for integrating the generated three-dimensional data and creating a virtual three-dimensional space that can be experienced by a user.

[0072] "Means for providing an interface" refers to the operating screen or device that allows the user to access and experience the generated virtual reality space.

[0073] "Metadata" refers to additional information associated with a photo or video, such as the location and date it was taken.

[0074] A "prompt" is an instruction given to a generative AI model to perform a specific task.

[0075] "Means for generating 3D models" refers to a structure or method for creating a three-dimensional digital object using metadata and photographic and video data.

[0076] This invention provides a system that converts photo data and video data into three-dimensional data and generates a virtual reality space based on that data. The system of the present invention is composed of three main elements: a user, a terminal, and a server.

[0077] First, users can select photos and videos from their devices and upload them to the system. In addition, users can enter metadata such as the location and date of the photos. For example, a user can select photos and videos from a "Family Vacation" folder and enter "Hawaii" as the location.

[0078] The device sends the selected file and the entered metadata to the server, where it verifies that the photo data (e.g., JPEG file) or video data (e.g., MP4 file) is in the appropriate format and then transfers the data to the server.

[0079] The server stores the received photo and video data, as well as metadata. Using a generative AI model, the server converts the photo and video data into three-dimensional data. At this point, the server inputs a prompt into the generative AI model to generate the three-dimensional data. For example, the prompt could be, "This photo was taken on a family trip on a beach in Hawaii. Please generate a 3D model of the person in the photo and create a virtual reality space that recreates the Hawaiian background."

[0080] The generative AI model analyzes photo data to create a 3D model. Similarly, it analyzes video data frame by frame to generate a dynamic 3D model. This allows for the creation of 3D digital objects of family and friends.

[0081] Furthermore, the server can generate a 3D model of the shooting location based on the metadata. For example, if the metadata "Hawaii" is entered, a 3D model of Hawaii's scenery and tourist attractions will be generated. This will create a virtual reality space.

[0082] The server saves the generated 3D data and virtual reality space in a form accessible to the user and generates a preview link. By providing this link to the user, the user can check the 3D data and virtual reality space on their own device. The user can immerse themselves in the virtual reality space using VR goggles or similar devices and experience it within.

[0083] As a specific example, if a user selects photos and videos from a "Family Trip" folder and enters metadata about Hawaii, the system will operate as follows: The user uploads the files from their device to the server, which receives and stores the data. The generative AI then converts the photos and videos into 3D images, generating a 3D model of Hawaii and constructing a virtual reality space. The user can then access the 3D data and virtual reality space and share the immersive experience with family and friends through VR goggles.

[0084] This system allows users to easily create 3D images of traditional 2D photos and videos, and recreate them in a realistic way, deepening interactions with family and friends and allowing them to share realistic experiences with family members who live far away.

[0085] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0086] Step 1:

[0087] Users select photos and video files from their devices and upload them to the system. They can also enter metadata (such as the location and date of the photo). The input data is a JPEG photo file or an MP4 video file. The output is the selected file and the entered metadata.

[0088] Step 2:

[0089] The device sends the file selected by the user and the entered metadata to the server. The device first checks the file format to ensure it is in the appropriate format (e.g., JPEG or MP4). It then transfers this data to the server. The input is the file and metadata, and the output is a confirmation of successful transfer to the server.

[0090] Step 3:

[0091] The server receives and stores the photo data, video data, and metadata sent from the device. The storage destination is internal storage or cloud storage (e.g., S3 bucket). The input is the data sent from the device, and the output is the path to the saved file.

[0092] Step 4:

[0093] The server uses a generative AI model to convert photo and video data into 3D data. During this process, the server generates a prompt and inputs it into the generative AI. An example of a prompt is, "This photo was taken on a family trip on a beach in Hawaii. Please generate 3D models of the people in the photo and create a virtual reality space that recreates the Hawaiian background." The server passes this prompt to the AI ​​model and receives the resulting 3D data. The inputs are photo data, video data, and the prompt, and the output is 3D data.

[0094] Step 5:

[0095] The server generates a 3D model of the filming location based on the metadata. For example, if the metadata includes "Hawaii," the server generates a 3D model of Hawaii's scenery and tourist attractions. The server searches and selects this from an existing 3D library, or generates a new one as needed. The input is the metadata, and the output is a 3D model of the filming location.

[0096] Step 6:

[0097] The server integrates the generated 3D data with a 3D model of the shooting location to construct a virtual reality space. This virtual reality space is then saved as a single dataset so that it can be experienced by the user. The input is the 3D data and the 3D model of the shooting location, and the output is a dataset of the integrated virtual reality space.

[0098] Step 7:

[0099] The server generates an access link to the generated virtual reality space and provides it to the user. For example, the server generates a preview link and sends it to the user's email address. The input is a dataset of the virtual reality space, and the output is a preview link to the user.

[0100] Step 8:

[0101] Users click the provided preview link to view the 3D data and virtual reality space on their own devices. Furthermore, users can use devices such as VR goggles to immerse themselves in the virtual reality space. The input is the preview link, and the output is the experience of the virtual reality space.

[0102] Through these steps, the system is able to create three-dimensional images of the user's photos and videos, recreating them in a realistic way.

[0103] (Application example 1)

[0104] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0105] Conventional photo and video viewing is two-dimensional, making it difficult for users to experience a sense of realism. Furthermore, when converting a physical store into a virtual reality space, there is also the problem of it being cumbersome to check detailed product information. Therefore, there is a demand for a system that allows users to check detailed product information without actually visiting the store, providing an experience that makes them feel as if they are in the actual store.

[0106] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0107] In this invention, the server includes means for receiving photo data and video data, means for using artificial intelligence to convert the photo data and video data into three-dimensional data, means for generating a virtual reality space from the three-dimensional data, means for providing an interface for users to access and experience the virtual reality space, and means for providing an interactive function for users to check detailed product information in the virtual reality space. This allows users to easily check detailed product information in the virtual reality space without visiting a physical store, and to experience the product with a sense of realism as if they were in the physical store.

[0108] "Photo data" refers to still image data taken with electronic devices such as digital cameras and smartphones.

[0109] "Moving image data" refers to dynamic video data consisting of a series of still images.

[0110] "Three-dimensional data" is data that expresses the position and shape of an object or space in three-dimensional space.

[0111] "Generative AI" is AI that can generate new data based on input data.

[0112] A "virtual reality space" is a virtual three-dimensional space generated by a computer, allowing users to experience the feeling of being inside that space.

[0113] An "interface" is a means of communication between a user and a system.

[0114] An "interactive function" is a function that provides an immediate response to the user's operations on the system.

[0115] To practice the invention, the following procedures and system configurations are used.

[0116] First, users upload photos and video data of the interior of a physical store and products to the system using a device such as a smartphone or tablet. The user selects the photos and video data and enters metadata such as the shooting location and product information.

[0117] The device sends the selected file and the entered metadata to the server. The server receives and stores the photo and video data. Based on the received data, a generative AI model is used to convert the 2D photos and videos into 3D data. This generative AI model can use services such as Google® Cloud AutoML Vision and Amazon Rekognition.

[0118] Based on the converted 3D data, the server uses a virtual reality space creation platform such as Unity 3D or Unreal Engine to generate a virtual reality space. The virtual reality space incorporates interactive functions such as the interior design of the actual store and detailed product information. This allows users to move freely through the virtual store using devices such as smartphones, head-mounted displays (HMDs), and smart glasses, and when they touch a product, detailed information about the product will pop up.

[0119] The generated virtual reality space is saved as a preview link online. Users can access this preview link and explore the store in the virtual reality space. This allows users to experience the store in a realistic way without visiting the actual store.

[0120] As a concrete example, consider a case where a store manager takes photos and product videos of the store and uploads them to the system. The metadata entered at this time could include, for example, a "new product section" or a "special sale area."

[0121] The generative AI model uses the following prompt:

[0122] 3D photo

[0123] "Generate a 3D model of a shop interior based on a 2D photo. Get the photo from the following link: [URL]"

[0124] 3D video

[0125] "Based on the uploaded video, please generate a 3D model of the product and 3D data that reproduces its movement. Please obtain the video data from the following link: [URL]"

[0126] Virtual reality store space

[0127] "Generate a virtual store environment based on the following metadata: Store name, location, product category, etc."

[0128] Finally, users can access a virtual reality space and experience an immersive store experience, allowing them to view detailed product information and make purchasing decisions without visiting a physical store.

[0129] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0130] Step 1:

[0131] Users upload photos and video data of store interiors and products from devices such as smartphones and tablets to the system. The input includes the photo and video data, as well as metadata such as the shooting location and product information, and the uploaded files are generated as output. The user operates the device to open the photo or video folder, select the file, and click the upload button.

[0132] Step 2:

[0133] The terminal sends the selected file and the entered metadata to the server. The terminal creates an HTTP request to send the file and sends it to the server. The input is the file and metadata selected by the user, and the output is the data sent to the server.

[0134] Step 3:

[0135] The server receives and stores photo and video data. It creates a directory structure to store the file data it receives in storage and writes the files. The data received from the device is input, and the saved file is generated as output.

[0136] Step 4:

[0137] Based on the received data, the server uses a generative AI model to convert photos and videos into 3D data. The server then sends API requests to Google Cloud AutoML Vision or Amazon Rekognition, analyzes the 2D data, and generates 3D data. The input is stored photo and video data, and the output is 3D data.

[0138] Step 5:

[0139] The server generates a virtual reality space based on the converted 3D data. The server reads the 3D data using the SDK of Unity 3D or Unreal Engine and performs the operation of constructing the virtual reality space programmatically. The generated 3D data is the input, and the virtual reality space is generated as the output.

[0140] Step 6:

[0141] The server implements interactive functions in the virtual reality space, allowing users to check detailed product information. The server sets up pop-up information and interaction events for each product's 3D model. The input is the virtual reality space and product information, and the output is a virtual reality space with interactive functions implemented.

[0142] Step 7:

[0143] The generated virtual reality space is saved as a link that can be previewed online. The server calls the link generation module to create a preview URL and saves it in the database. The input is the virtual reality space, and the output is a preview link.

[0144] Step 8:

[0145] Users can access the virtual reality space using the preview link and get a realistic store experience. Users access the link using a smartphone or head-mounted display and explore the virtual reality space. The preview link is the input, and access to the virtual reality space is possible as the output.

[0146] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0147] The present invention relates to a system that converts photo data and video data into three-dimensional data and generates a virtual reality space based on that data. One of the features of the present invention is that it can provide a more personalized virtual reality experience by combining it with an emotion engine that recognizes the user's emotions.

[0148] First, the user selects and uploads family photos and videos from their device. Users can optionally enter metadata such as the shooting location and date and time. For example, users can select photos and videos from the "Family Trip" folder and enter "Hawaii" as the shooting location.

[0149] Next, the device sends the selected file data and the entered metadata to the server. The server receives and stores the file data sent by the user. Based on the received file data, the server uses a generative AI to convert the photo and video data into 3D data. Specifically, the generative AI analyzes the 2D photos and generates a 3D model of the family. It also generates moving 3D data for the video data.

[0150] Furthermore, the server generates a 3D model of the shooting location based on the metadata as needed. For example, if "Hawaii" is entered as metadata, a 3D model recreating the scenery and tourist spots of Hawaii is generated. A virtual reality space is then constructed based on this 3D model.

[0151] This system adds a function in which the emotion engine recognizes the user's emotions and dynamically adjusts the virtual reality space based on that information. When a user accesses the virtual reality space, the emotion engine recognizes the user's current emotional state from their facial expressions, voice, and movements. For example, if the user is smiling, the system will determine that this is "fun."

[0152] The server analyzes the user's emotional data recognized by the emotion engine and provides the user with an appropriate virtual reality experience. For example, if the user is sad, the system will provide a virtual space with bright colors and cheerful music to help the user recall happy memories. On the other hand, if the user looks happy, the system will provide a virtual reality experience with a fast tempo that matches the user's emotions.

[0153] As a specific example, let's consider what happens when a user selects a photo from a "Family Trip" folder and enters metadata about Hawaii. The user first uploads the file from their device to the server, which receives and stores the data. The generative AI then converts the photos and videos into 3D images, generating a 3D model of Hawaii to create a virtual reality space. The user then accesses the 3D data and virtual reality space, sharing an immersive experience with family and friends through VR goggles. During this process, the emotion engine recognizes the user's emotions and dynamically adjusts the settings of the virtual space based on that information.

[0154] As a result, the present invention provides users with a more personalized, realistic, and immersive re-experience of their memories, and also serves as an effective means for deepening interactions with family and friends.

[0155] The processing flow will be explained below.

[0156] Step 1:

[0157] The user opens the "Family Trip" folder on their device, selects the photo IMG_001.jpg and the video VID_001.mp4, and presses the upload button. If necessary, the user can enter metadata such as "Hawaii" as the shooting location and "July 2023" as the date and time.

[0158] Step 2:

[0159] The terminal transmits the user's selected photo data and video data, as well as the input metadata, to the server, which receives the request and stores the file data and metadata.

[0160] Step 3:

[0161] The server sends the saved photo data IMG_001.jpg and video data VID_001.mp4 to the generation AI, which analyzes the photo data and video data and converts them into three-dimensional data.

[0162] Step 4:

[0163] The server analyzes the metadata and generates a 3D model of "Hawaii," the location specified for the shoot. The generated 3D data is combined with the 3D model of the shooting location to create a virtual reality space.

[0164] Step 5:

[0165] The server stores the generated 3D data and virtual reality space, and generates a preview link that the user can access. The user receives this preview link and checks it on their own device.

[0166] Step 6:

[0167] The user clicks the preview link to view the 3D images and videos. The device downloads the 3D data from the server and displays it in the browser.

[0168] Step 7:

[0169] The user puts on the VR goggles and logs in to experience the generated virtual reality space. In the virtual reality space, the user can enjoy an immersive experience.

[0170] Step 8:

[0171] The emotion engine recognizes the user's emotional state in real time from their facial expressions, voice, and actions. For example, if the user is smiling, the emotion engine will determine that they are "happy."

[0172] Step 9:

[0173] The server analyzes the emotion data sent from the emotion engine and dynamically adjusts the virtual reality space based on that information. For example, if the user is recognized as "happy," it will add bright colors and cheerful music to the virtual space.

[0174] Step 10:

[0175] The device sends an invitation link to allow the user to invite other family members or friends to the virtual reality space, and the family or friends who receive the link also log in to the virtual reality space and share the experience with the user.

[0176] Through these steps, users can transform traditional flat photos and videos into 3D images, enjoy personalized virtual reality experiences based on emotions, and deepen interactions with family and friends, even sharing realistic experiences with family members who live far away.

[0177] Example 2

[0178] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0179] While conventional systems could create 3D images of photo and video data, they were unable to dynamically adjust the virtual reality space to take into account the user's emotions. This made it difficult to provide a personalized experience for each user. Furthermore, their ability to generate 3D models of actual shooting locations was insufficient, limiting their ability to provide a more realistic and immersive experience.

[0180] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for receiving photo data and video data, a means for using a generation artificial intelligence to convert the photo data and video data into three-dimensional data, a means for generating a virtual reality space from the three-dimensional data, and a means for recognizing a user's emotions and dynamically adjusting the virtual reality space. This makes it possible to provide a personalized virtual reality experience according to the user's emotions, recreating the shooting location, and improving the sense of realism.

[0181] "Photo data" refers to still image data, which are digital images saved in formats such as JPEG or PNG.

[0182] "Video data" refers to data containing dynamic visual information consisting of a series of image frames, and is digital video saved in formats such as MP4 or AVI.

[0183] "Three-dimensional data" is data that contains three-dimensional information about an object or space, and is digital data that includes coordinate information on the X, Y, and Z axes.

[0184] A "virtual reality space" is a computer-generated three-dimensional environment that a user can experience visually and acoustically.

[0185] "Generative AI" is an artificial intelligence technology that mimics human intellectual processes and generates new information and models from given data.

[0186] An "interface" is a means for exchanging information between a user and a system, and includes GUIs (graphical user interfaces) and VR (virtual reality) goggles.

[0187] "Metadata" refers to additional information associated with a photo or video, such as the location, date, and time of the photo, and tags.

[0188] An "emotion recognition engine" is software or hardware that analyzes a user's facial expressions, voice, movements, etc., and determines their emotional state.

[0189] "Dynamic adjustment" refers to changing system settings and behavior in real time in response to changing conditions or environments.

[0190] "Storage means" refers to devices or methods for temporarily or permanently retaining data, including databases and cloud storage.

[0191] This invention is a system that converts photo and video data into three-dimensional data and generates a virtual reality space based on that data. In particular, by combining this with technology that recognizes the user's emotions and dynamically adjusts the virtual reality space, it is possible to provide a more personalized virtual reality experience.

[0192] A user can use their own device to select family photos and videos and upload them to the system. For example, the user selects photos and videos from a "Family Trip" folder and enters metadata such as "Hawaii" as the location where the photos were taken. The device then sends the selected files (photos and videos) and the entered metadata to the server using an HTTP request.

[0193] The server receives the file data and metadata sent by the user and stores them in cloud storage (e.g., Amazon S3). After saving, it generates a file identifier and records it in a database. The server then uses generative artificial intelligence (e.g., DALL-E 3 by OpenAI (registered trademark)) to convert the photo and video data into three-dimensional data. Specifically, the server inputs the photo along with a prompt into the generative AI, which generates a three-dimensional model. Video data is also analyzed in the same way and converted into moving three-dimensional data.

[0194] The server then generates a 3D model of the location based on the metadata. For example, if the metadata includes "Hawaii," the server uses a generative AI model to generate a 3D model of Hawaii's scenery and tourist attractions. This forms the framework for the virtual reality space.

[0195] When a user accesses a virtual reality space, an emotion engine (e.g., Affectiva) recognizes the user's emotions. The emotion engine determines the user's current emotional state from their facial expressions, voice, and movements, and sends this data to the server. The server analyzes the emotion recognition data and dynamically adjusts the colors and music of the virtual reality space according to the user's emotions. For example, if the user is in an emotional state of "fun," the virtual space will be set up with bright colors and cheerful music.

[0196] Finally, the server provides the user with a tailored virtual reality space. Using VR goggles, the user can experience three-dimensional family photos and videos, as well as a reconstructed location (e.g., Hawaii) in real time. The emotion engine continuously monitors the user's emotions and adjusts the virtual space accordingly.

[0197] As a specific example, let's consider the process when a user selects photos from a "Family Trip" folder and enters metadata for Hawaii. The user uploads photos and videos from their device, and the server receives and stores the data. The generative artificial intelligence (DALL-E 3) converts the photos and videos into 3D images and generates a 3D model of Hawaii to create a virtual reality space. The user accesses the 3D data and virtual reality space and shares an immersive experience with family and friends through VR goggles. During this process, the emotion engine recognizes the user's emotions and dynamically adjusts the settings of the virtual space based on that information.

[0198] Examples of prompt sentences that can be used include:

[0199] plain

[0200] Convert this family photo into a three-dimensional model.

[0201] Recreate Hawaii's tourist attractions as a three-dimensional model.

[0202] As a result, the present invention provides users with a more personalized, realistic, and immersive re-experience of their memories, and also serves as an effective means for deepening interactions with family and friends.

[0203] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0204] Step 1: Select and upload your data

[0205] Subject: User

[0206] Description: A user selects family photos and videos from their device and uploads them to the system. For example, the user selects photos and videos from a "Family Vacation" folder and enters the metadata, such as "Hawaii," as the location where the photos were taken.

[0207] Input: Photo data, video data, metadata (shooting location, date and time, etc.)

[0208] Output: Selected files and metadata are kept on your device

[0209] Specific behavior: Displays a file selection dialog and preserves the selected file and any entered metadata.

[0210] Step 2: Sending data

[0211] Subject: Terminal

[0212] Description: The terminal sends the selected file and the entered metadata to the server. The file is sent using an HTTP request.

[0213] Input: Selected files, metadata

[0214] Output: Notification of completion of transmission to the server

[0215] Specific behavior: Generates an HTTP POST request and sends the file and metadata to the server.

[0216] Step 3: Save the file data

[0217] Subject: Server

[0218] Description: The server receives the file data and metadata sent by the user and stores them in cloud storage. After storing them, it generates a file identifier and records it in the database.

[0219] Input: Submitted file data, metadata

[0220] Output: Saved to cloud storage and recorded in database

[0221] What it does: Upload a file to cloud storage, generate a file identifier, and then store the identifier and metadata in a database.

[0222] Step 4: Convert to 3D data

[0223] Subject: Server

[0224] Description: The server calls the generative AI model to analyze stored photo and video data. It inputs photo data along with a prompt to generate a three-dimensional model. Video data is similarly analyzed and converted into moving three-dimensional data.

[0225] Input: Saved photo data, video data, prompt (e.g., "Convert this family photo into a 3D model.")

[0226] Output: 3D data (3D models and moving models)

[0227] Specific operation: Call a generation AI (e.g., DALL-E 3), input a photo and a prompt, and generate a 3D model. Similarly, analyze video data to generate dynamic 3D data.

[0228] Step 5: Generate a 3D model of the location

[0229] Subject: Server

[0230] Description: The server generates a 3D model of the shooting location based on the metadata. For example, if the metadata includes "Hawaii," it generates a 3D model that recreates the scenery and tourist attractions of Hawaii.

[0231] Input: Metadata (location)

[0232] Output: 3D model of the location

[0233] What it does: Uses a generative AI model to generate a 3D model based on metadata (e.g., "Please recreate Hawaii's tourist attractions as a 3D, three-dimensional model.").

[0234] Step 6: Emotion Recognition

[0235] Subject: Emotion Engine

[0236] Description: When a user accesses a virtual reality space, the emotion engine recognizes the user's emotions. The emotion engine determines the user's current emotional state from their facial expressions, voice, and movements, and sends that data to the server.

[0237] Input: User's facial expression data, voice data, and movement data

[0238] Output: Emotional state data

[0239] Specific behavior: An emotion engine (e.g., Affectiva) analyzes the user's real-time data to determine their emotional state.

[0240] Step 7: Adjust based on sentiment data

[0241] Subject: Server

[0242] Description: The server analyzes data from the emotion engine and dynamically adjusts the settings of the virtual reality space according to the user's emotions. For example, if the user is in a "happy" emotional state, the virtual space will be set to bright colors and cheerful music.

[0243] Input: Emotional state data

[0244] Output: Calibrated virtual reality space configuration

[0245] Specific operation: The server analyzes the emotional data and changes the settings of the virtual reality space, such as color, music, and environment.

[0246] Step 8: Deliver the virtual reality experience

[0247] Subject: Server

[0248] Description: The server finally provides the user with a tailored virtual reality space. The user can wear VR goggles and experience family photos and videos in 3D, along with a 3D model of the shooting location, for an immersive experience.

[0249] Input: Calibrated virtual reality space

[0250] Output: User's virtual reality experience

[0251] Specific operation: The user puts on VR goggles, accesses a virtual reality space in real time, and experiences three-dimensional data and a coordinated environment.

[0252] (Application example 2)

[0253] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0254] Conventional virtual reality systems are unable to personalize the experience based on the user's emotions, and are limited to providing a uniform experience to users. In addition, in certain applications such as shopping experiences, personalization that dynamically reflects the user's interests and emotions is required, but this has been difficult to achieve.

[0255] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0256] In this invention, the server includes means for receiving photo data and video data, means for using a generation artificial intelligence to convert the photo data and video data into three-dimensional data, means for generating a virtual reality space from the three-dimensional data, means for providing an interface for a user to access and experience the virtual reality space, and means for using an emotion engine to recognize the user's emotions and dynamically adjust the virtual reality space. This enables the virtual reality space to be personalized based on the user's emotions, enabling individual suggestions and displays that reflect the user's interests and emotions, particularly in a shopping experience.

[0257] "Photo data" refers to data of still images taken by the user.

[0258] "Video data" refers to video data captured by the user.

[0259] "Three-dimensional data" is three-dimensional data generated based on photographic data and video data.

[0260] "Generative AI" is an AI that analyzes photo data and video data and generates three-dimensional data.

[0261] "Virtual reality space" is a virtual three-dimensional space generated based on three-dimensional data.

[0262] An "interface" is a device or software that allows a user to access and experience a virtual reality space.

[0263] The "emotion engine" is an engine that recognizes the user's emotions and dynamically adjusts the virtual reality space based on those emotions.

[0264] "Metadata" is additional information about photo data and video data, including the location and date of the photo.

[0265] The present invention relates to a technology for recognizing a user's emotions and dynamically adjusting a virtual reality space based on the emotions. Specific examples are given below.

[0266] First, the user selects and uploads family photos and videos from their device (e.g., smartphone or smart glasses). If necessary, the user can also enter metadata such as the shooting location and date and time. For example, the user can select photos and videos from the "Family Trip" folder and enter "Hawaii" as the shooting location. The device then sends the selected file data and the entered metadata to the server.

[0267] The server performs data processing and calculations using a series of hardware and software such as the following:

[0268] Generative AI model: The server uses a generative AI model to convert the received photo and video data into 3D data. Specifically, it analyzes 2D photos to generate a 3D model, and generates dynamic 3D data for video data.

[0269] Virtual reality environment tools: Based on the generated 3D data, a virtual reality space is constructed using virtual reality environment tools (e.g., Unity, Unreal Engine). In particular, a three-dimensional model of the shooting location is generated based on the metadata, and the model is incorporated into the virtual reality space.

[0270] Emotion engine: When a user accesses a virtual reality space, the emotion engine (e.g., Amazon Rekognition, Microsoft® Emotion API) recognizes the user's emotional state from their facial expressions and voice. Based on this data, the server adjusts the VR environment in real time.

[0271] For example, if a user uploads photos from a "family trip" and specifies "Hawaii" as the location, the server receives and stores the data. The generative AI model then converts the photo and video data into 3D, generating a 3D model of Hawaii to create a virtual reality space. When the user then accesses the virtual reality space through smart glasses, the emotion engine recognizes the user's emotional state and, for example, provides a VR experience using bright colors and cheerful music if the user is having fun.

[0272] Below is an example of a prompt sentence:

[0273] Example prompt sentence:

[0274] "A user uploads a family photo. Use a generative AI model to convert the photo into 3D data and build a virtual reality space. Then, use an emotion engine to recognize the user's emotions and personalize the product displays in the virtual store based on those emotions."

[0275] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0276] Step 1:

[0277] Users select and upload family photos and videos from their own devices. The input includes photo data, video data, and metadata (such as the location and date of the photo). Based on this, the device processes the data by sending the file data and metadata to the server. The output is the photo data, video data, and metadata sent to the server.

[0278] Step 2:

[0279] The server receives and stores file data and metadata sent by users. The input is the photo data, video data, and metadata sent by users, and performs data operations to store it in storage. The output is the stored file data and metadata.

[0280] Step 3:

[0281] The server inputs the stored photo and video data into a generative AI model to generate 3D data. The input is the photo and video data stored in storage, and data processing is performed to convert the 2D data into 3D data. The output is 3D data.

[0282] Step 4:

[0283] The server generates a virtual reality space based on the 3D data. The input is the generated 3D data, which is then passed to a virtual reality environment tool for data calculations to construct a VR space. The output is the completed virtual reality space.

[0284] Step 5:

[0285] To recognize the user's emotions, the emotion engine analyzes their facial expressions and voice when they access the virtual reality space through a VR interface (e.g., smart glasses). The input is the user's facial expressions and voice data, which the emotion engine analyzes and processes. The output is the user's emotional state data.

[0286] Step 6:

[0287] The server receives emotional state data from the emotion engine and dynamically adjusts the virtual reality space based on that data. The input is the emotional state data from the emotion engine, and it performs data calculations to change the settings of the VR space based on that data. The output is a personalized virtual reality experience.

[0288] As a specific example, if a user uploads photos from a "family trip" and specifies "Hawaii" as the location, the data is sent from the device to the server in step 1, and the server stores them in step 2. In step 3, a generative AI model converts the photo and video data into 3D, and in step 4, a virtual reality space containing a 3D model of Hawaii is generated. Next, in step 5, an emotion engine analyzes the user's emotions, and in step 6, a personalized VR experience using bright colors and cheerful music is provided.

[0289] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0290] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0291] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0292] [Second embodiment]

[0293] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0294] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0295] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0296] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0297] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0298] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0299] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0300] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0301] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0302] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0303] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0304] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0305] This invention provides a system that converts photo data and video data into three-dimensional data and generates a virtual reality space based on that data. The system of the present invention is composed of three main elements: a user, a terminal, and a server.

[0306] First, the user selects and uploads family photos and videos from their device. The user can optionally enter metadata such as the shooting location and date and time. For example, the user can select photos and videos from the "Family Trip" folder and enter "Hawaii" as the shooting location.

[0307] Next, the device sends the file selected by the user and the metadata entered to the server. The server receives and stores the file data sent by the user. Based on the received file data, the server uses generative AI to convert the photo and video data into 3D data. For example, generative AI analyzes 2D photos and generates a 3D model of a family. It also does the same for video data, generating moving 3D data.

[0308] The server also generates a 3D model of the shooting location based on the metadata as needed. For example, if "Hawaii" is entered as metadata, a 3D model recreating the scenery and tourist spots of Hawaii is generated. A virtual reality space is then constructed based on this 3D model.

[0309] The server then saves the generated 3D data and virtual reality space in a user-accessible state and generates a preview link. The user can access the generated 3D data and virtual reality space and view it on their own device. For example, the user can click the preview link to view the 3D image or video in a browser.

[0310] Furthermore, users can use VR goggles to immerse themselves in the virtual reality space and relive their past experiences. They can also invite other family members and friends to the virtual reality space and relive shared memories in an immersive way. For example, a user can send an invitation link to their family members, allowing them to explore a virtual Hawaiian space together and share memories.

[0311] This system allows users to easily turn conventional flat photos and videos into three dimensions, allowing them to relive memories in a more immersive way. It also has the advantage of deepening interactions with family and friends, allowing them to share realistic experiences with family members who live far away.

[0312] For example, if a user selects photos and videos from a "Family Trip" folder and enters metadata about Hawaii, the system works as follows: The user first uploads the files from their device to the server, which receives and stores the data. The generative AI then creates a 3D model of Hawaii and constructs a virtual reality space. The user can then access the 3D data and virtual reality space and share the immersive experience with family and friends through VR goggles.

[0313] As described above, the present invention provides users with a means for reliving memories in a more realistic and convenient manner.

[0314] The processing flow will be explained below.

[0315] Step 1:

[0316] The user selects photos and videos from their device and presses the upload button. If necessary, the user can enter metadata such as the shooting location and date and time.

[0317] Step 2:

[0318] The terminal sends the selected file data and input metadata to the server, which receives the request and stores the file data and metadata.

[0319] Step 3:

[0320] The server sends the stored photo and video data to the generation AI, which analyzes the 2D data and generates a corresponding 3D model. Through this process, the generation AI outputs the three-dimensional data.

[0321] Step 4:

[0322] The server analyzes the metadata and generates a 3D model of the location where the photo was taken, if necessary. For example, if the metadata includes "Hawaii," it generates a 3D model of Hawaiian scenery and tourist attractions.

[0323] Step 5:

[0324] The server combines the generated 3D data with the virtual reality space and stores it as data for each user. It also generates a preview link so that the user can access this data.

[0325] Step 6:

[0326] Users click the preview link to view the 3D image or video. The device downloads the 3D data from the server and displays it in the browser.

[0327] Step 7:

[0328] Users put on VR goggles and log in to the virtual reality space, where they can enjoy an immersive experience with their family and friends.

[0329] Step 8:

[0330] The user invites other family members and friends to the virtual reality space, and the device sends an invitation link to the family and friends, allowing everyone to access the virtual reality space.

[0331] The above is the specific flow of the program's processing, which allows the user to re-experience memories in a three-dimensional and realistic way.

[0332] Example 1

[0333] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0334] Conventional photos and videos are two-dimensional, and the visual information they provide is limited, making it difficult for users to feel a sense of presence or a deep experience. Furthermore, the means for sharing those experiences with family and friends who live far away are also limited. This makes it difficult to recreate memories in a more realistic and immersive way, and to deepen shared experiences.

[0335] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0336] In this invention, the server includes means for receiving photo data and video data, means for using artificial intelligence to convert the photo data and video data into three-dimensional data, means for generating a virtual reality space based on the three-dimensional data and metadata, and means for providing an interface for users to access and experience the virtual reality space. This allows users to convert two-dimensional photos and videos into three-dimensional space, re-create memories in an immersive way, and share realistic experiences with family and friends who live far away.

[0337] "Photo data and video data" refers to digital files of still images and videos taken by users.

[0338] "Means for receiving" refers to the structure or method for transferring and storing digital files from a user's terminal to a server.

[0339] "Means using generative artificial intelligence" refers to a structure or method for converting two-dimensional photographs or videos into three-dimensional data using artificial intelligence techniques.

[0340] "Three-dimensional data" refers to a three-dimensional digital object created based on two-dimensional information from photographs and videos.

[0341] "Means for generating a virtual reality space" refers to a structure or method for integrating the generated three-dimensional data and creating a virtual three-dimensional space that can be experienced by a user.

[0342] "Means for providing an interface" refers to the operating screen or device that allows the user to access and experience the generated virtual reality space.

[0343] "Metadata" refers to additional information associated with a photo or video, such as the location and date it was taken.

[0344] A "prompt" is an instruction given to a generative AI model to perform a specific task.

[0345] "Means for generating 3D models" refers to a structure or method for creating a three-dimensional digital object using metadata and photographic and video data.

[0346] This invention provides a system that converts photo data and video data into three-dimensional data and generates a virtual reality space based on that data. The system of the present invention is composed of three main elements: a user, a terminal, and a server.

[0347] First, users can select photos and videos from their devices and upload them to the system. In addition, users can enter metadata such as the location and date of the photos. For example, a user can select photos and videos from a "Family Vacation" folder and enter "Hawaii" as the location.

[0348] The device sends the selected file and the entered metadata to the server, where it verifies that the photo data (e.g., JPEG file) or video data (e.g., MP4 file) is in the appropriate format and then transfers the data to the server.

[0349] The server stores the received photo and video data, as well as metadata. Using a generative AI model, the server converts the photo and video data into three-dimensional data. At this point, the server inputs a prompt into the generative AI model to generate the three-dimensional data. For example, the prompt could be, "This photo was taken on a family trip on a beach in Hawaii. Please generate a 3D model of the person in the photo and create a virtual reality space that recreates the Hawaiian background."

[0350] The generative AI model analyzes photo data to create a 3D model. Similarly, it analyzes video data frame by frame to generate a dynamic 3D model. This allows for the creation of 3D digital objects of family and friends.

[0351] Furthermore, the server can generate a 3D model of the shooting location based on the metadata. For example, if the metadata "Hawaii" is entered, a 3D model of Hawaii's scenery and tourist attractions will be generated. This will create a virtual reality space.

[0352] The server saves the generated 3D data and virtual reality space in a form accessible to the user and generates a preview link. By providing this link to the user, the user can check the 3D data and virtual reality space on their own device. The user can immerse themselves in the virtual reality space using VR goggles or similar devices and experience it within.

[0353] As a specific example, if a user selects photos and videos from a "Family Trip" folder and enters metadata about Hawaii, the system will operate as follows: The user uploads the files from their device to the server, which receives and stores the data. The generative AI then converts the photos and videos into 3D images, generating a 3D model of Hawaii and constructing a virtual reality space. The user can then access the 3D data and virtual reality space and share the immersive experience with family and friends through VR goggles.

[0354] This system allows users to easily create 3D images of traditional 2D photos and videos, and recreate them in a realistic way, deepening interactions with family and friends and allowing them to share realistic experiences with family members who live far away.

[0355] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0356] Step 1:

[0357] Users select photos and video files from their devices and upload them to the system. They can also enter metadata (such as the location and date of the photo). The input data is a JPEG photo file or an MP4 video file. The output is the selected file and the entered metadata.

[0358] Step 2:

[0359] The device sends the file selected by the user and the entered metadata to the server. The device first checks the file format to ensure it is in the appropriate format (e.g., JPEG or MP4). It then transfers this data to the server. The input is the file and metadata, and the output is a confirmation of successful transfer to the server.

[0360] Step 3:

[0361] The server receives and stores the photo data, video data, and metadata sent from the device. The storage destination is internal storage or cloud storage (e.g., S3 bucket). The input is the data sent from the device, and the output is the path to the saved file.

[0362] Step 4:

[0363] The server uses a generative AI model to convert photo and video data into 3D data. During this process, the server generates a prompt and inputs it into the generative AI. An example of a prompt is, "This photo was taken on a family trip on a beach in Hawaii. Please generate 3D models of the people in the photo and create a virtual reality space that recreates the Hawaiian background." The server passes this prompt to the AI ​​model and receives the resulting 3D data. The inputs are photo data, video data, and the prompt, and the output is 3D data.

[0364] Step 5:

[0365] The server generates a 3D model of the filming location based on the metadata. For example, if the metadata includes "Hawaii," the server generates a 3D model of Hawaii's scenery and tourist attractions. The server searches and selects this from an existing 3D library, or generates a new one as needed. The input is the metadata, and the output is a 3D model of the filming location.

[0366] Step 6:

[0367] The server integrates the generated 3D data with a 3D model of the shooting location to construct a virtual reality space. This virtual reality space is then saved as a single dataset so that it can be experienced by the user. The input is the 3D data and the 3D model of the shooting location, and the output is a dataset of the integrated virtual reality space.

[0368] Step 7:

[0369] The server generates an access link to the generated virtual reality space and provides it to the user. For example, the server generates a preview link and sends it to the user's email address. The input is a dataset of the virtual reality space, and the output is a preview link to the user.

[0370] Step 8:

[0371] Users click the provided preview link to view the 3D data and virtual reality space on their own devices. Furthermore, users can use devices such as VR goggles to immerse themselves in the virtual reality space. The input is the preview link, and the output is the experience of the virtual reality space.

[0372] Through these steps, the system is able to create three-dimensional images of the user's photos and videos, recreating them in a realistic way.

[0373] (Application example 1)

[0374] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0375] Conventional photo and video viewing is two-dimensional, making it difficult for users to experience a sense of realism. Furthermore, when converting a physical store into a virtual reality space, there is also the problem of it being cumbersome to check detailed product information. Therefore, there is a demand for a system that allows users to check detailed product information without actually visiting the store, providing an experience that makes them feel as if they are in the actual store.

[0376] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0377] In this invention, the server includes means for receiving photo data and video data, means for using artificial intelligence to convert the photo data and video data into three-dimensional data, means for generating a virtual reality space from the three-dimensional data, means for providing an interface for users to access and experience the virtual reality space, and means for providing an interactive function for users to check detailed product information in the virtual reality space. This allows users to easily check detailed product information in the virtual reality space without visiting a physical store, and to experience the product with a sense of realism as if they were in the physical store.

[0378] "Photo data" refers to still image data taken with electronic devices such as digital cameras and smartphones.

[0379] "Moving image data" refers to dynamic video data consisting of a series of still images.

[0380] "Three-dimensional data" is data that expresses the position and shape of an object or space in three-dimensional space.

[0381] "Generative AI" is AI that can generate new data based on input data.

[0382] A "virtual reality space" is a virtual three-dimensional space generated by a computer, allowing users to experience the feeling of being inside that space.

[0383] An "interface" is a means of communication between a user and a system.

[0384] An "interactive function" is a function that provides an immediate response to the user's operations on the system.

[0385] To practice the invention, the following procedures and system configurations are used.

[0386] First, users upload photos and video data of the interior of a physical store and products to the system using a device such as a smartphone or tablet. The user selects the photos and video data and enters metadata such as the shooting location and product information.

[0387] The device sends the selected file and the entered metadata to the server. The server receives and stores the photo and video data. Based on the received data, a generative AI model is used to convert the 2D photos and videos into 3D data. This generative AI model can use services such as Google Cloud AutoML Vision and Amazon Rekognition.

[0388] Based on the converted 3D data, the server uses a virtual reality space creation platform such as Unity 3D or Unreal Engine to generate a virtual reality space. The virtual reality space incorporates interactive functions such as the interior design of the actual store and detailed product information. This allows users to move freely through the virtual store using devices such as smartphones, head-mounted displays (HMDs), and smart glasses, and when they touch a product, detailed information about the product will pop up.

[0389] The generated virtual reality space is saved as a preview link online. Users can access this preview link and explore the store in the virtual reality space. This allows users to experience the store in a realistic way without visiting the actual store.

[0390] As a concrete example, consider a case where a store manager takes photos and product videos of the store and uploads them to the system. The metadata entered at this time could include, for example, a "new product section" or a "special sale area."

[0391] The generative AI model uses the following prompt:

[0392] 3D photo

[0393] "Generate a 3D model of a shop interior based on a 2D photo. Get the photo from the following link: [URL]"

[0394] 3D video

[0395] "Based on the uploaded video, please generate a 3D model of the product and 3D data that reproduces its movement. Please obtain the video data from the following link: [URL]"

[0396] Virtual reality store space

[0397] "Generate a virtual store environment based on the following metadata: Store name, location, product category, etc."

[0398] Finally, users can access a virtual reality space and experience an immersive store experience, allowing them to view detailed product information and make purchasing decisions without visiting a physical store.

[0399] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0400] Step 1:

[0401] Users upload photos and video data of store interiors and products from devices such as smartphones and tablets to the system. The input includes the photo and video data, as well as metadata such as the shooting location and product information, and the uploaded files are generated as output. The user operates the device to open the photo or video folder, select the file, and click the upload button.

[0402] Step 2:

[0403] The terminal sends the selected file and the entered metadata to the server. The terminal creates an HTTP request to send the file and sends it to the server. The input is the file and metadata selected by the user, and the output is the data sent to the server.

[0404] Step 3:

[0405] The server receives and stores photo and video data. It creates a directory structure to store the file data it receives in storage and writes the files. The data received from the device is input, and the saved file is generated as output.

[0406] Step 4:

[0407] Based on the received data, the server uses a generative AI model to convert photos and videos into 3D data. The server then sends API requests to Google Cloud AutoML Vision or Amazon Rekognition, analyzes the 2D data, and generates 3D data. The input is stored photo and video data, and the output is 3D data.

[0408] Step 5:

[0409] The server generates a virtual reality space based on the converted 3D data. The server reads the 3D data using the SDK of Unity 3D or Unreal Engine and performs the operation of constructing the virtual reality space programmatically. The generated 3D data is the input, and the virtual reality space is generated as the output.

[0410] Step 6:

[0411] The server implements interactive functions in the virtual reality space, allowing users to check detailed product information. The server sets up pop-up information and interaction events for each product's 3D model. The input is the virtual reality space and product information, and the output is a virtual reality space with interactive functions implemented.

[0412] Step 7:

[0413] The generated virtual reality space is saved as a link that can be previewed online. The server calls the link generation module to create a preview URL and saves it in the database. The input is the virtual reality space, and the output is a preview link.

[0414] Step 8:

[0415] Users can access the virtual reality space using the preview link and get a realistic store experience. Users access the link using a smartphone or head-mounted display and explore the virtual reality space. The preview link is the input, and access to the virtual reality space is possible as the output.

[0416] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0417] The present invention relates to a system that converts photo data and video data into three-dimensional data and generates a virtual reality space based on that data. One of the features of the present invention is that it can provide a more personalized virtual reality experience by combining it with an emotion engine that recognizes the user's emotions.

[0418] First, the user selects and uploads family photos and videos from their device. Users can optionally enter metadata such as the shooting location and date and time. For example, users can select photos and videos from the "Family Trip" folder and enter "Hawaii" as the shooting location.

[0419] Next, the device sends the selected file data and the entered metadata to the server. The server receives and stores the file data sent by the user. Based on the received file data, the server uses a generative AI to convert the photo and video data into 3D data. Specifically, the generative AI analyzes the 2D photos and generates a 3D model of the family. It also generates moving 3D data for the video data.

[0420] Furthermore, the server generates a 3D model of the shooting location based on the metadata as needed. For example, if "Hawaii" is entered as metadata, a 3D model recreating the scenery and tourist spots of Hawaii is generated. A virtual reality space is then constructed based on this 3D model.

[0421] This system adds a function in which the emotion engine recognizes the user's emotions and dynamically adjusts the virtual reality space based on that information. When a user accesses the virtual reality space, the emotion engine recognizes the user's current emotional state from their facial expressions, voice, and movements. For example, if the user is smiling, the system will determine that this is "fun."

[0422] The server analyzes the user's emotional data recognized by the emotion engine and provides the user with an appropriate virtual reality experience. For example, if the user is sad, the system will provide a virtual space with bright colors and cheerful music to help the user recall happy memories. On the other hand, if the user looks happy, the system will provide a virtual reality experience with a fast tempo that matches the user's emotions.

[0423] As a specific example, let's consider what happens when a user selects a photo from a "Family Trip" folder and enters metadata about Hawaii. The user first uploads the file from their device to the server, which receives and stores the data. The generative AI then converts the photos and videos into 3D images, generating a 3D model of Hawaii to create a virtual reality space. The user then accesses the 3D data and virtual reality space, sharing an immersive experience with family and friends through VR goggles. During this process, the emotion engine recognizes the user's emotions and dynamically adjusts the settings of the virtual space based on that information.

[0424] As a result, the present invention provides users with a more personalized, realistic, and immersive re-experience of their memories, and also serves as an effective means for deepening interactions with family and friends.

[0425] The processing flow will be explained below.

[0426] Step 1:

[0427] The user opens the "Family Trip" folder on their device, selects the photo IMG_001.jpg and the video VID_001.mp4, and presses the upload button. If necessary, the user can enter metadata such as "Hawaii" as the shooting location and "July 2023" as the date and time.

[0428] Step 2:

[0429] The terminal transmits the user's selected photo data and video data, as well as the input metadata, to the server, which receives the request and stores the file data and metadata.

[0430] Step 3:

[0431] The server sends the saved photo data IMG_001.jpg and video data VID_001.mp4 to the generation AI, which analyzes the photo data and video data and converts them into three-dimensional data.

[0432] Step 4:

[0433] The server analyzes the metadata and generates a 3D model of "Hawaii," the location specified for the shoot. The generated 3D data is combined with the 3D model of the shooting location to create a virtual reality space.

[0434] Step 5:

[0435] The server stores the generated 3D data and virtual reality space, and generates a preview link that the user can access. The user receives this preview link and checks it on their own device.

[0436] Step 6:

[0437] The user clicks the preview link to view the 3D images and videos. The device downloads the 3D data from the server and displays it in the browser.

[0438] Step 7:

[0439] The user puts on the VR goggles and logs in to experience the generated virtual reality space. In the virtual reality space, the user can enjoy an immersive experience.

[0440] Step 8:

[0441] The emotion engine recognizes the user's emotional state in real time from their facial expressions, voice, and actions. For example, if the user is smiling, the emotion engine will determine that they are "happy."

[0442] Step 9:

[0443] The server analyzes the emotion data sent from the emotion engine and dynamically adjusts the virtual reality space based on that information. For example, if the user is recognized as "happy," it will add bright colors and cheerful music to the virtual space.

[0444] Step 10:

[0445] The device sends an invitation link to allow the user to invite other family members or friends to the virtual reality space, and the family or friends who receive the link also log in to the virtual reality space and share the experience with the user.

[0446] Through these steps, users can transform traditional flat photos and videos into 3D images, enjoy personalized virtual reality experiences based on emotions, and deepen interactions with family and friends, even sharing realistic experiences with family members who live far away.

[0447] Example 2

[0448] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0449] While conventional systems could create 3D images of photo and video data, they were unable to dynamically adjust the virtual reality space to take into account the user's emotions. This made it difficult to provide a personalized experience for each user. Furthermore, their ability to generate 3D models of actual shooting locations was insufficient, limiting their ability to provide a more realistic and immersive experience.

[0450] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for receiving photo data and video data, a means for using a generation artificial intelligence to convert the photo data and video data into three-dimensional data, a means for generating a virtual reality space from the three-dimensional data, and a means for recognizing a user's emotions and dynamically adjusting the virtual reality space. This makes it possible to provide a personalized virtual reality experience according to the user's emotions, recreating the shooting location, and improving the sense of realism.

[0451] "Photo data" refers to still image data, which are digital images saved in formats such as JPEG or PNG.

[0452] "Video data" refers to data containing dynamic visual information consisting of a series of image frames, and is digital video saved in formats such as MP4 or AVI.

[0453] "Three-dimensional data" is data that contains three-dimensional information about an object or space, and is digital data that includes coordinate information on the X, Y, and Z axes.

[0454] A "virtual reality space" is a computer-generated three-dimensional environment that a user can experience visually and acoustically.

[0455] "Generative AI" is an artificial intelligence technology that mimics human intellectual processes and generates new information and models from given data.

[0456] An "interface" is a means for exchanging information between a user and a system, and includes GUIs (graphical user interfaces) and VR (virtual reality) goggles.

[0457] "Metadata" refers to additional information associated with a photo or video, such as the location, date, and time of the photo, and tags.

[0458] An "emotion recognition engine" is software or hardware that analyzes a user's facial expressions, voice, movements, etc., and determines their emotional state.

[0459] "Dynamic adjustment" refers to changing system settings and behavior in real time in response to changing conditions or environments.

[0460] "Storage means" refers to devices or methods for temporarily or permanently retaining data, including databases and cloud storage.

[0461] This invention is a system that converts photo and video data into three-dimensional data and generates a virtual reality space based on that data. In particular, by combining this with technology that recognizes the user's emotions and dynamically adjusts the virtual reality space, it is possible to provide a more personalized virtual reality experience.

[0462] A user can use their own device to select family photos and videos and upload them to the system. For example, the user selects photos and videos from a "Family Trip" folder and enters metadata such as "Hawaii" as the location where the photos were taken. The device then sends the selected files (photos and videos) and the entered metadata to the server using an HTTP request.

[0463] The server receives the file data and metadata sent by the user and stores them in cloud storage (e.g., Amazon S3). After saving, it generates a file identifier and records it in a database. The server then uses generative artificial intelligence (e.g., OpenAI's DALL-E 3) to convert the photo and video data into three-dimensional data. Specifically, the photo is input into the generative AI along with a prompt sentence, which generates a three-dimensional three-dimensional model. Video data is also analyzed in the same way and converted into moving three-dimensional data.

[0464] The server then generates a 3D model of the location based on the metadata. For example, if the metadata includes "Hawaii," the server uses a generative AI model to generate a 3D model of Hawaii's scenery and tourist attractions. This forms the framework for the virtual reality space.

[0465] When a user accesses a virtual reality space, an emotion engine (e.g., Affectiva) recognizes the user's emotions. The emotion engine determines the user's current emotional state from their facial expressions, voice, and movements, and sends this data to the server. The server analyzes the emotion recognition data and dynamically adjusts the colors and music of the virtual reality space according to the user's emotions. For example, if the user is in an emotional state of "fun," the virtual space will be set up with bright colors and cheerful music.

[0466] Finally, the server provides the user with a tailored virtual reality space. Using VR goggles, the user can experience three-dimensional family photos and videos, as well as a reconstructed location (e.g., Hawaii) in real time. The emotion engine continuously monitors the user's emotions and adjusts the virtual space accordingly.

[0467] As a specific example, let's consider the process when a user selects photos from a "Family Trip" folder and enters metadata for Hawaii. The user uploads photos and videos from their device, and the server receives and stores the data. The generative artificial intelligence (DALL-E 3) converts the photos and videos into 3D images and generates a 3D model of Hawaii to create a virtual reality space. The user accesses the 3D data and virtual reality space and shares an immersive experience with family and friends through VR goggles. During this process, the emotion engine recognizes the user's emotions and dynamically adjusts the settings of the virtual space based on that information.

[0468] Examples of prompt sentences that can be used include:

[0469] plain

[0470] Convert this family photo into a three-dimensional model.

[0471] Recreate Hawaii's tourist attractions as a three-dimensional model.

[0472] As a result, the present invention provides users with a more personalized, realistic, and immersive re-experience of their memories, and also serves as an effective means for deepening interactions with family and friends.

[0473] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0474] Step 1: Select and upload your data

[0475] Subject: User

[0476] Description: A user selects family photos and videos from their device and uploads them to the system. For example, the user selects photos and videos from a "Family Vacation" folder and enters the metadata, such as "Hawaii," as the location where the photos were taken.

[0477] Input: Photo data, video data, metadata (shooting location, date and time, etc.)

[0478] Output: Selected files and metadata are kept on your device

[0479] Specific behavior: Displays a file selection dialog and preserves the selected file and any entered metadata.

[0480] Step 2: Sending data

[0481] Subject: Terminal

[0482] Description: The terminal sends the selected file and the entered metadata to the server. The file is sent using an HTTP request.

[0483] Input: Selected files, metadata

[0484] Output: Notification of completion of transmission to the server

[0485] Specific behavior: Generates an HTTP POST request and sends the file and metadata to the server.

[0486] Step 3: Save the file data

[0487] Subject: Server

[0488] Description: The server receives the file data and metadata sent by the user and stores them in cloud storage. After storing them, it generates a file identifier and records it in the database.

[0489] Input: Submitted file data, metadata

[0490] Output: Saved to cloud storage and recorded in database

[0491] What it does: Upload a file to cloud storage, generate a file identifier, and then store the identifier and metadata in a database.

[0492] Step 4: Convert to 3D data

[0493] Subject: Server

[0494] Description: The server calls the generative AI model to analyze stored photo and video data. It inputs photo data along with a prompt to generate a three-dimensional model. Video data is similarly analyzed and converted into moving three-dimensional data.

[0495] Input: Saved photo data, video data, prompt (e.g., "Convert this family photo into a 3D model.")

[0496] Output: 3D data (3D models and moving models)

[0497] Specific operation: Call a generation AI (e.g., DALL-E 3), input a photo and a prompt, and generate a 3D model. Similarly, analyze video data to generate dynamic 3D data.

[0498] Step 5: Generate a 3D model of the location

[0499] Subject: Server

[0500] Description: The server generates a 3D model of the shooting location based on the metadata. For example, if the metadata includes "Hawaii," it generates a 3D model that recreates the scenery and tourist attractions of Hawaii.

[0501] Input: Metadata (location)

[0502] Output: 3D model of the location

[0503] What it does: Uses a generative AI model to generate a 3D model based on metadata (e.g., "Please recreate Hawaii's tourist attractions as a 3D, three-dimensional model.").

[0504] Step 6: Emotion Recognition

[0505] Subject: Emotion Engine

[0506] Description: When a user accesses a virtual reality space, the emotion engine recognizes the user's emotions. The emotion engine determines the user's current emotional state from their facial expressions, voice, and movements, and sends that data to the server.

[0507] Input: User's facial expression data, voice data, and movement data

[0508] Output: Emotional state data

[0509] Specific behavior: An emotion engine (e.g., Affectiva) analyzes the user's real-time data to determine their emotional state.

[0510] Step 7: Adjust based on sentiment data

[0511] Subject: Server

[0512] Description: The server analyzes data from the emotion engine and dynamically adjusts the settings of the virtual reality space according to the user's emotions. For example, if the user is in a "happy" emotional state, the virtual space will be set to bright colors and cheerful music.

[0513] Input: Emotional state data

[0514] Output: Calibrated virtual reality space configuration

[0515] Specific operation: The server analyzes the emotional data and changes the settings of the virtual reality space, such as color, music, and environment.

[0516] Step 8: Deliver the virtual reality experience

[0517] Subject: Server

[0518] Description: The server finally provides the user with a tailored virtual reality space. The user can wear VR goggles and experience family photos and videos in 3D, along with a 3D model of the shooting location, for an immersive experience.

[0519] Input: Calibrated virtual reality space

[0520] Output: User's virtual reality experience

[0521] Specific operation: The user puts on VR goggles, accesses a virtual reality space in real time, and experiences three-dimensional data and a coordinated environment.

[0522] (Application example 2)

[0523] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0524] Conventional virtual reality systems are unable to personalize the experience based on the user's emotions, and are limited to providing a uniform experience to users. In addition, in certain applications such as shopping experiences, personalization that dynamically reflects the user's interests and emotions is required, but this has been difficult to achieve.

[0525] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0526] In this invention, the server includes means for receiving photo data and video data, means for using a generation artificial intelligence to convert the photo data and video data into three-dimensional data, means for generating a virtual reality space from the three-dimensional data, means for providing an interface for a user to access and experience the virtual reality space, and means for using an emotion engine to recognize the user's emotions and dynamically adjust the virtual reality space. This enables the virtual reality space to be personalized based on the user's emotions, enabling individual suggestions and displays that reflect the user's interests and emotions, particularly in a shopping experience.

[0527] "Photo data" refers to data of still images taken by the user.

[0528] "Video data" refers to video data captured by the user.

[0529] "Three-dimensional data" is three-dimensional data generated based on photographic data and video data.

[0530] "Generative AI" is an AI that analyzes photo data and video data and generates three-dimensional data.

[0531] "Virtual reality space" is a virtual three-dimensional space generated based on three-dimensional data.

[0532] An "interface" is a device or software that allows a user to access and experience a virtual reality space.

[0533] The "emotion engine" is an engine that recognizes the user's emotions and dynamically adjusts the virtual reality space based on those emotions.

[0534] "Metadata" is additional information about photo data and video data, including the location and date of the photo.

[0535] The present invention relates to a technology for recognizing a user's emotions and dynamically adjusting a virtual reality space based on the emotions. Specific examples are given below.

[0536] First, the user selects and uploads family photos and videos from their device (e.g., smartphone or smart glasses). If necessary, the user can also enter metadata such as the shooting location and date and time. For example, the user can select photos and videos from the "Family Trip" folder and enter "Hawaii" as the shooting location. The device then sends the selected file data and the entered metadata to the server.

[0537] The server performs data processing and calculations using a series of hardware and software such as the following:

[0538] Generative AI model: The server uses a generative AI model to convert the received photo and video data into 3D data. Specifically, it analyzes 2D photos to generate a 3D model, and generates dynamic 3D data for video data.

[0539] Virtual reality environment tools: Based on the generated 3D data, a virtual reality space is constructed using virtual reality environment tools (e.g., Unity, Unreal Engine). In particular, a three-dimensional model of the shooting location is generated based on the metadata, and the model is incorporated into the virtual reality space.

[0540] Emotion engine: When a user accesses a virtual reality space, an emotion engine (e.g., Amazon Rekognition, Microsoft Emotion API) recognizes the user's emotional state from their facial expressions and voice. Based on this data, the server adjusts the VR environment in real time.

[0541] For example, if a user uploads photos from a "family trip" and specifies "Hawaii" as the location, the server receives and stores the data. The generative AI model then converts the photo and video data into 3D, generating a 3D model of Hawaii to create a virtual reality space. When the user then accesses the virtual reality space through smart glasses, the emotion engine recognizes the user's emotional state and, for example, provides a VR experience using bright colors and cheerful music if the user is having fun.

[0542] Below is an example of a prompt sentence:

[0543] Example prompt sentence:

[0544] "A user uploads a family photo. Use a generative AI model to convert the photo into 3D data and build a virtual reality space. Then, use an emotion engine to recognize the user's emotions and personalize the product displays in the virtual store based on those emotions."

[0545] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0546] Step 1:

[0547] Users select and upload family photos and videos from their own devices. The input includes photo data, video data, and metadata (such as the location and date of the photo). Based on this, the device processes the data by sending the file data and metadata to the server. The output is the photo data, video data, and metadata sent to the server.

[0548] Step 2:

[0549] The server receives and stores file data and metadata sent by users. The input is the photo data, video data, and metadata sent by users, and performs data operations to store it in storage. The output is the stored file data and metadata.

[0550] Step 3:

[0551] The server inputs the stored photo and video data into a generative AI model to generate 3D data. The input is the photo and video data stored in storage, and data processing is performed to convert the 2D data into 3D data. The output is 3D data.

[0552] Step 4:

[0553] The server generates a virtual reality space based on the 3D data. The input is the generated 3D data, which is then passed to a virtual reality environment tool for data calculations to construct a VR space. The output is the completed virtual reality space.

[0554] Step 5:

[0555] To recognize the user's emotions, the emotion engine analyzes their facial expressions and voice when they access the virtual reality space through a VR interface (e.g., smart glasses). The input is the user's facial expressions and voice data, which the emotion engine analyzes and processes. The output is the user's emotional state data.

[0556] Step 6:

[0557] The server receives emotional state data from the emotion engine and dynamically adjusts the virtual reality space based on that data. The input is the emotional state data from the emotion engine, and it performs data calculations to change the settings of the VR space based on that data. The output is a personalized virtual reality experience.

[0558] As a specific example, if a user uploads photos from a "family trip" and specifies "Hawaii" as the location, the data is sent from the device to the server in step 1, and the server stores them in step 2. In step 3, a generative AI model converts the photo and video data into 3D, and in step 4, a virtual reality space containing a 3D model of Hawaii is generated. Next, in step 5, an emotion engine analyzes the user's emotions, and in step 6, a personalized VR experience using bright colors and cheerful music is provided.

[0559] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0560] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0561] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0562] [Third embodiment]

[0563] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0564] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0565] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0566] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0567] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0568] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0569] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0570] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0571] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0572] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0573] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0574] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0575] This invention provides a system that converts photo data and video data into three-dimensional data and generates a virtual reality space based on that data. The system of the present invention is composed of three main elements: a user, a terminal, and a server.

[0576] First, the user selects and uploads family photos and videos from their device. The user can optionally enter metadata such as the shooting location and date and time. For example, the user can select photos and videos from the "Family Trip" folder and enter "Hawaii" as the shooting location.

[0577] Next, the device sends the file selected by the user and the metadata entered to the server. The server receives and stores the file data sent by the user. Based on the received file data, the server uses generative AI to convert the photo and video data into 3D data. For example, generative AI analyzes 2D photos and generates a 3D model of a family. It also does the same for video data, generating moving 3D data.

[0578] The server also generates a 3D model of the shooting location based on the metadata as needed. For example, if "Hawaii" is entered as metadata, a 3D model recreating the scenery and tourist spots of Hawaii is generated. A virtual reality space is then constructed based on this 3D model.

[0579] The server then saves the generated 3D data and virtual reality space in a user-accessible state and generates a preview link. The user can access the generated 3D data and virtual reality space and view it on their own device. For example, the user can click the preview link to view the 3D image or video in a browser.

[0580] Furthermore, users can use VR goggles to immerse themselves in the virtual reality space and relive their past experiences. They can also invite other family members and friends to the virtual reality space and relive shared memories in an immersive way. For example, a user can send an invitation link to their family members, allowing them to explore a virtual Hawaiian space together and share memories.

[0581] This system allows users to easily turn conventional flat photos and videos into three dimensions, allowing them to relive memories in a more immersive way. It also has the advantage of deepening interactions with family and friends, allowing them to share realistic experiences with family members who live far away.

[0582] For example, if a user selects photos and videos from a "Family Trip" folder and enters metadata about Hawaii, the system works as follows: The user first uploads the files from their device to the server, which receives and stores the data. The generative AI then creates a 3D model of Hawaii and constructs a virtual reality space. The user can then access the 3D data and virtual reality space and share the immersive experience with family and friends through VR goggles.

[0583] As described above, the present invention provides users with a means for reliving memories in a more realistic and convenient manner.

[0584] The processing flow will be explained below.

[0585] Step 1:

[0586] The user selects photos and videos from their device and presses the upload button. If necessary, the user can enter metadata such as the shooting location and date and time.

[0587] Step 2:

[0588] The terminal sends the selected file data and input metadata to the server, which receives the request and stores the file data and metadata.

[0589] Step 3:

[0590] The server sends the stored photo and video data to the generation AI, which analyzes the 2D data and generates a corresponding 3D model. Through this process, the generation AI outputs the three-dimensional data.

[0591] Step 4:

[0592] The server analyzes the metadata and generates a 3D model of the location where the photo was taken, if necessary. For example, if the metadata includes "Hawaii," it generates a 3D model of Hawaiian scenery and tourist attractions.

[0593] Step 5:

[0594] The server combines the generated 3D data with the virtual reality space and stores it as data for each user. It also generates a preview link so that the user can access this data.

[0595] Step 6:

[0596] Users click the preview link to view the 3D image or video. The device downloads the 3D data from the server and displays it in the browser.

[0597] Step 7:

[0598] Users put on VR goggles and log in to the virtual reality space, where they can enjoy an immersive experience with their family and friends.

[0599] Step 8:

[0600] The user invites other family members and friends to the virtual reality space, and the device sends an invitation link to the family and friends, allowing everyone to access the virtual reality space.

[0601] The above is the specific flow of the program's processing, which allows the user to re-experience memories in a three-dimensional and realistic way.

[0602] Example 1

[0603] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0604] Conventional photos and videos are two-dimensional, and the visual information they provide is limited, making it difficult for users to feel a sense of presence or a deep experience. Furthermore, the means for sharing those experiences with family and friends who live far away are also limited. This makes it difficult to recreate memories in a more realistic and immersive way, and to deepen shared experiences.

[0605] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0606] In this invention, the server includes means for receiving photo data and video data, means for using artificial intelligence to convert the photo data and video data into three-dimensional data, means for generating a virtual reality space based on the three-dimensional data and metadata, and means for providing an interface for users to access and experience the virtual reality space. This allows users to convert two-dimensional photos and videos into three-dimensional space, re-create memories in an immersive way, and share realistic experiences with family and friends who live far away.

[0607] "Photo data and video data" refers to digital files of still images and videos taken by users.

[0608] "Means for receiving" refers to the structure or method for transferring and storing digital files from a user's terminal to a server.

[0609] "Means using generative artificial intelligence" refers to a structure or method for converting two-dimensional photographs or videos into three-dimensional data using artificial intelligence techniques.

[0610] "Three-dimensional data" refers to a three-dimensional digital object created based on two-dimensional information from photographs and videos.

[0611] "Means for generating a virtual reality space" refers to a structure or method for integrating the generated three-dimensional data and creating a virtual three-dimensional space that can be experienced by a user.

[0612] "Means for providing an interface" refers to the operating screen or device that allows the user to access and experience the generated virtual reality space.

[0613] "Metadata" refers to additional information associated with a photo or video, such as the location and date it was taken.

[0614] A "prompt" is an instruction given to a generative AI model to perform a specific task.

[0615] "Means for generating 3D models" refers to a structure or method for creating a three-dimensional digital object using metadata and photographic and video data.

[0616] This invention provides a system that converts photo data and video data into three-dimensional data and generates a virtual reality space based on that data. The system of the present invention is composed of three main elements: a user, a terminal, and a server.

[0617] First, users can select photos and videos from their devices and upload them to the system. In addition, users can enter metadata such as the location and date of the photos. For example, a user can select photos and videos from a "Family Vacation" folder and enter "Hawaii" as the location.

[0618] The device sends the selected file and the entered metadata to the server, where it verifies that the photo data (e.g., JPEG file) or video data (e.g., MP4 file) is in the appropriate format and then transfers the data to the server.

[0619] The server stores the received photo and video data, as well as metadata. Using a generative AI model, the server converts the photo and video data into three-dimensional data. At this point, the server inputs a prompt into the generative AI model to generate the three-dimensional data. For example, the prompt could be, "This photo was taken on a family trip on a beach in Hawaii. Please generate a 3D model of the person in the photo and create a virtual reality space that recreates the Hawaiian background."

[0620] The generative AI model analyzes photo data to create a 3D model. Similarly, it analyzes video data frame by frame to generate a dynamic 3D model. This allows for the creation of 3D digital objects of family and friends.

[0621] Furthermore, the server can generate a 3D model of the shooting location based on the metadata. For example, if the metadata "Hawaii" is entered, a 3D model of Hawaii's scenery and tourist attractions will be generated. This will create a virtual reality space.

[0622] The server saves the generated 3D data and virtual reality space in a form accessible to the user and generates a preview link. By providing this link to the user, the user can check the 3D data and virtual reality space on their own device. The user can immerse themselves in the virtual reality space using VR goggles or similar devices and experience it within.

[0623] As a specific example, if a user selects photos and videos from a "Family Trip" folder and enters metadata about Hawaii, the system will operate as follows: The user uploads the files from their device to the server, which receives and stores the data. The generative AI then converts the photos and videos into 3D images, generating a 3D model of Hawaii and constructing a virtual reality space. The user can then access the 3D data and virtual reality space and share the immersive experience with family and friends through VR goggles.

[0624] This system allows users to easily create 3D images of traditional 2D photos and videos, and recreate them in a realistic way, deepening interactions with family and friends and allowing them to share realistic experiences with family members who live far away.

[0625] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0626] Step 1:

[0627] Users select photos and video files from their devices and upload them to the system. They can also enter metadata (such as the location and date of the photo). The input data is a JPEG photo file or an MP4 video file. The output is the selected file and the entered metadata.

[0628] Step 2:

[0629] The device sends the file selected by the user and the entered metadata to the server. The device first checks the file format to ensure it is in the appropriate format (e.g., JPEG or MP4). It then transfers this data to the server. The input is the file and metadata, and the output is a confirmation of successful transfer to the server.

[0630] Step 3:

[0631] The server receives and stores the photo data, video data, and metadata sent from the device. The storage destination is internal storage or cloud storage (e.g., S3 bucket). The input is the data sent from the device, and the output is the path to the saved file.

[0632] Step 4:

[0633] The server uses a generative AI model to convert photo and video data into 3D data. During this process, the server generates a prompt and inputs it into the generative AI. An example of a prompt is, "This photo was taken on a family trip on a beach in Hawaii. Please generate 3D models of the people in the photo and create a virtual reality space that recreates the Hawaiian background." The server passes this prompt to the AI ​​model and receives the resulting 3D data. The inputs are photo data, video data, and the prompt, and the output is 3D data.

[0634] Step 5:

[0635] The server generates a 3D model of the filming location based on the metadata. For example, if the metadata includes "Hawaii," the server generates a 3D model of Hawaii's scenery and tourist attractions. The server searches and selects this from an existing 3D library, or generates a new one as needed. The input is the metadata, and the output is a 3D model of the filming location.

[0636] Step 6:

[0637] The server integrates the generated 3D data with a 3D model of the shooting location to construct a virtual reality space. This virtual reality space is then saved as a single dataset so that it can be experienced by the user. The input is the 3D data and the 3D model of the shooting location, and the output is a dataset of the integrated virtual reality space.

[0638] Step 7:

[0639] The server generates an access link to the generated virtual reality space and provides it to the user. For example, the server generates a preview link and sends it to the user's email address. The input is a dataset of the virtual reality space, and the output is a preview link to the user.

[0640] Step 8:

[0641] Users click the provided preview link to view the 3D data and virtual reality space on their own devices. Furthermore, users can use devices such as VR goggles to immerse themselves in the virtual reality space. The input is the preview link, and the output is the experience of the virtual reality space.

[0642] Through these steps, the system is able to create three-dimensional images of the user's photos and videos, recreating them in a realistic way.

[0643] (Application example 1)

[0644] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0645] Conventional photo and video viewing is two-dimensional, making it difficult for users to experience a sense of realism. Furthermore, when converting a physical store into a virtual reality space, there is also the problem of it being cumbersome to check detailed product information. Therefore, there is a demand for a system that allows users to check detailed product information without actually visiting the store, providing an experience that makes them feel as if they are in the actual store.

[0646] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0647] In this invention, the server includes means for receiving photo data and video data, means for using artificial intelligence to convert the photo data and video data into three-dimensional data, means for generating a virtual reality space from the three-dimensional data, means for providing an interface for users to access and experience the virtual reality space, and means for providing an interactive function for users to check detailed product information in the virtual reality space. This allows users to easily check detailed product information in the virtual reality space without visiting a physical store, and to experience the product with a sense of realism as if they were in the physical store.

[0648] "Photo data" refers to still image data taken with electronic devices such as digital cameras and smartphones.

[0649] "Moving image data" refers to dynamic video data consisting of a series of still images.

[0650] "Three-dimensional data" is data that expresses the position and shape of an object or space in three-dimensional space.

[0651] "Generative AI" is AI that can generate new data based on input data.

[0652] A "virtual reality space" is a virtual three-dimensional space generated by a computer, allowing users to experience the feeling of being inside that space.

[0653] An "interface" is a means of communication between a user and a system.

[0654] An "interactive function" is a function that provides an immediate response to the user's operations on the system.

[0655] To practice the invention, the following procedures and system configurations are used.

[0656] First, users upload photos and video data of the interior of a physical store and products to the system using a device such as a smartphone or tablet. The user selects the photos and video data and enters metadata such as the shooting location and product information.

[0657] The device sends the selected file and the entered metadata to the server. The server receives and stores the photo and video data. Based on the received data, a generative AI model is used to convert the 2D photos and videos into 3D data. This generative AI model can use services such as Google Cloud AutoML Vision and Amazon Rekognition.

[0658] Based on the converted 3D data, the server uses a virtual reality space creation platform such as Unity 3D or Unreal Engine to generate a virtual reality space. The virtual reality space incorporates interactive functions such as the interior design of the actual store and detailed product information. This allows users to move freely through the virtual store using devices such as smartphones, head-mounted displays (HMDs), and smart glasses, and when they touch a product, detailed information about the product will pop up.

[0659] The generated virtual reality space is saved as a preview link online. Users can access this preview link and explore the store in the virtual reality space. This allows users to experience the store in a realistic way without visiting the actual store.

[0660] As a concrete example, consider a case where a store manager takes photos and product videos of the store and uploads them to the system. The metadata entered at this time could include, for example, a "new product section" or a "special sale area."

[0661] The generative AI model uses the following prompt:

[0662] 3D photo

[0663] "Generate a 3D model of a shop interior based on a 2D photo. Get the photo from the following link: [URL]"

[0664] 3D video

[0665] "Based on the uploaded video, please generate a 3D model of the product and 3D data that reproduces its movement. Please obtain the video data from the following link: [URL]"

[0666] Virtual reality store space

[0667] "Generate a virtual store environment based on the following metadata: Store name, location, product category, etc."

[0668] Finally, users can access a virtual reality space and experience an immersive store experience, allowing them to view detailed product information and make purchasing decisions without visiting a physical store.

[0669] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0670] Step 1:

[0671] Users upload photos and video data of store interiors and products from devices such as smartphones and tablets to the system. The input includes the photo and video data, as well as metadata such as the shooting location and product information, and the uploaded files are generated as output. The user operates the device to open the photo or video folder, select the file, and click the upload button.

[0672] Step 2:

[0673] The terminal sends the selected file and the entered metadata to the server. The terminal creates an HTTP request to send the file and sends it to the server. The input is the file and metadata selected by the user, and the output is the data sent to the server.

[0674] Step 3:

[0675] The server receives and stores photo and video data. It creates a directory structure to store the file data it receives in storage and writes the files. The data received from the device is input, and the saved file is generated as output.

[0676] Step 4:

[0677] Based on the received data, the server uses a generative AI model to convert photos and videos into 3D data. The server then sends API requests to Google Cloud AutoML Vision or Amazon Rekognition, analyzes the 2D data, and generates 3D data. The input is stored photo and video data, and the output is 3D data.

[0678] Step 5:

[0679] The server generates a virtual reality space based on the converted 3D data. The server reads the 3D data using the SDK of Unity 3D or Unreal Engine and performs the operation of constructing the virtual reality space programmatically. The generated 3D data is the input, and the virtual reality space is generated as the output.

[0680] Step 6:

[0681] The server implements interactive functions in the virtual reality space, allowing users to check detailed product information. The server sets up pop-up information and interaction events for each product's 3D model. The input is the virtual reality space and product information, and the output is a virtual reality space with interactive functions implemented.

[0682] Step 7:

[0683] The generated virtual reality space is saved as a link that can be previewed online. The server calls the link generation module to create a preview URL and saves it in the database. The input is the virtual reality space, and the output is a preview link.

[0684] Step 8:

[0685] Users can access the virtual reality space using the preview link and get a realistic store experience. Users access the link using a smartphone or head-mounted display and explore the virtual reality space. The preview link is the input, and access to the virtual reality space is possible as the output.

[0686] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0687] The present invention relates to a system that converts photo data and video data into three-dimensional data and generates a virtual reality space based on that data. One of the features of the present invention is that it can provide a more personalized virtual reality experience by combining it with an emotion engine that recognizes the user's emotions.

[0688] First, the user selects and uploads family photos and videos from their device. Users can optionally enter metadata such as the shooting location and date and time. For example, users can select photos and videos from the "Family Trip" folder and enter "Hawaii" as the shooting location.

[0689] Next, the device sends the selected file data and the entered metadata to the server. The server receives and stores the file data sent by the user. Based on the received file data, the server uses a generative AI to convert the photo and video data into 3D data. Specifically, the generative AI analyzes the 2D photos and generates a 3D model of the family. It also generates moving 3D data for the video data.

[0690] Furthermore, the server generates a 3D model of the shooting location based on the metadata as needed. For example, if "Hawaii" is entered as metadata, a 3D model recreating the scenery and tourist spots of Hawaii is generated. A virtual reality space is then constructed based on this 3D model.

[0691] This system adds a function in which the emotion engine recognizes the user's emotions and dynamically adjusts the virtual reality space based on that information. When a user accesses the virtual reality space, the emotion engine recognizes the user's current emotional state from their facial expressions, voice, and movements. For example, if the user is smiling, the system will determine that this is "fun."

[0692] The server analyzes the user's emotional data recognized by the emotion engine and provides the user with an appropriate virtual reality experience. For example, if the user is sad, the system will provide a virtual space with bright colors and cheerful music to help the user recall happy memories. On the other hand, if the user looks happy, the system will provide a virtual reality experience with a fast tempo that matches the user's emotions.

[0693] As a specific example, let's consider what happens when a user selects a photo from a "Family Trip" folder and enters metadata about Hawaii. The user first uploads the file from their device to the server, which receives and stores the data. The generative AI then converts the photos and videos into 3D images, generating a 3D model of Hawaii to create a virtual reality space. The user then accesses the 3D data and virtual reality space, sharing an immersive experience with family and friends through VR goggles. During this process, the emotion engine recognizes the user's emotions and dynamically adjusts the settings of the virtual space based on that information.

[0694] As a result, the present invention provides users with a more personalized, realistic, and immersive re-experience of their memories, and also serves as an effective means for deepening interactions with family and friends.

[0695] The processing flow will be explained below.

[0696] Step 1:

[0697] The user opens the "Family Trip" folder on their device, selects the photo IMG_001.jpg and the video VID_001.mp4, and presses the upload button. If necessary, the user can enter metadata such as "Hawaii" as the shooting location and "July 2023" as the date and time.

[0698] Step 2:

[0699] The terminal transmits the user's selected photo data and video data, as well as the input metadata, to the server, which receives the request and stores the file data and metadata.

[0700] Step 3:

[0701] The server sends the saved photo data IMG_001.jpg and video data VID_001.mp4 to the generation AI, which analyzes the photo data and video data and converts them into three-dimensional data.

[0702] Step 4:

[0703] The server analyzes the metadata and generates a 3D model of "Hawaii," the location specified for the shoot. The generated 3D data is combined with the 3D model of the shooting location to create a virtual reality space.

[0704] Step 5:

[0705] The server stores the generated 3D data and virtual reality space, and generates a preview link that the user can access. The user receives this preview link and checks it on their own device.

[0706] Step 6:

[0707] The user clicks the preview link to view the 3D images and videos. The device downloads the 3D data from the server and displays it in the browser.

[0708] Step 7:

[0709] The user puts on the VR goggles and logs in to experience the generated virtual reality space. In the virtual reality space, the user can enjoy an immersive experience.

[0710] Step 8:

[0711] The emotion engine recognizes the user's emotional state in real time from their facial expressions, voice, and actions. For example, if the user is smiling, the emotion engine will determine that they are "happy."

[0712] Step 9:

[0713] The server analyzes the emotion data sent from the emotion engine and dynamically adjusts the virtual reality space based on that information. For example, if the user is recognized as "happy," it will add bright colors and cheerful music to the virtual space.

[0714] Step 10:

[0715] The device sends an invitation link to allow the user to invite other family members or friends to the virtual reality space, and the family or friends who receive the link also log in to the virtual reality space and share the experience with the user.

[0716] Through these steps, users can transform traditional flat photos and videos into 3D images, enjoy personalized virtual reality experiences based on emotions, and deepen interactions with family and friends, even sharing realistic experiences with family members who live far away.

[0717] Example 2

[0718] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0719] While conventional systems could create 3D images of photo and video data, they were unable to dynamically adjust the virtual reality space to take into account the user's emotions. This made it difficult to provide a personalized experience for each user. Furthermore, their ability to generate 3D models of actual shooting locations was insufficient, limiting their ability to provide a more realistic and immersive experience.

[0720] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for receiving photo data and video data, a means for using a generation artificial intelligence to convert the photo data and video data into three-dimensional data, a means for generating a virtual reality space from the three-dimensional data, and a means for recognizing a user's emotions and dynamically adjusting the virtual reality space. This makes it possible to provide a personalized virtual reality experience according to the user's emotions, recreating the shooting location, and improving the sense of realism.

[0721] "Photo data" refers to still image data, which are digital images saved in formats such as JPEG or PNG.

[0722] "Video data" refers to data containing dynamic visual information consisting of a series of image frames, and is digital video saved in formats such as MP4 or AVI.

[0723] "Three-dimensional data" is data that contains three-dimensional information about an object or space, and is digital data that includes coordinate information on the X, Y, and Z axes.

[0724] A "virtual reality space" is a computer-generated three-dimensional environment that a user can experience visually and acoustically.

[0725] "Generative AI" is an artificial intelligence technology that mimics human intellectual processes and generates new information and models from given data.

[0726] An "interface" is a means for exchanging information between a user and a system, and includes GUIs (graphical user interfaces) and VR (virtual reality) goggles.

[0727] "Metadata" refers to additional information associated with a photo or video, such as the location, date, and time of the photo, and tags.

[0728] An "emotion recognition engine" is software or hardware that analyzes a user's facial expressions, voice, movements, etc., and determines their emotional state.

[0729] "Dynamic adjustment" refers to changing system settings and behavior in real time in response to changing conditions or environments.

[0730] "Storage means" refers to devices or methods for temporarily or permanently retaining data, including databases and cloud storage.

[0731] This invention is a system that converts photo and video data into three-dimensional data and generates a virtual reality space based on that data. In particular, by combining this with technology that recognizes the user's emotions and dynamically adjusts the virtual reality space, it is possible to provide a more personalized virtual reality experience.

[0732] A user can use their own device to select family photos and videos and upload them to the system. For example, the user selects photos and videos from a "Family Trip" folder and enters metadata such as "Hawaii" as the location where the photos were taken. The device then sends the selected files (photos and videos) and the entered metadata to the server using an HTTP request.

[0733] The server receives the file data and metadata sent by the user and stores them in cloud storage (e.g., Amazon S3). After saving, it generates a file identifier and records it in a database. The server then uses generative artificial intelligence (e.g., OpenAI's DALL-E 3) to convert the photo and video data into three-dimensional data. Specifically, the photo is input into the generative AI along with a prompt sentence, which generates a three-dimensional three-dimensional model. Video data is also analyzed in the same way and converted into moving three-dimensional data.

[0734] The server then generates a 3D model of the location based on the metadata. For example, if the metadata includes "Hawaii," the server uses a generative AI model to generate a 3D model of Hawaii's scenery and tourist attractions. This forms the framework for the virtual reality space.

[0735] When a user accesses a virtual reality space, an emotion engine (e.g., Affectiva) recognizes the user's emotions. The emotion engine determines the user's current emotional state from their facial expressions, voice, and movements, and sends this data to the server. The server analyzes the emotion recognition data and dynamically adjusts the colors and music of the virtual reality space according to the user's emotions. For example, if the user is in an emotional state of "fun," the virtual space will be set up with bright colors and cheerful music.

[0736] Finally, the server provides the user with a tailored virtual reality space. Using VR goggles, the user can experience three-dimensional family photos and videos, as well as a reconstructed location (e.g., Hawaii) in real time. The emotion engine continuously monitors the user's emotions and adjusts the virtual space accordingly.

[0737] As a specific example, let's consider the process when a user selects photos from a "Family Trip" folder and enters metadata for Hawaii. The user uploads photos and videos from their device, and the server receives and stores the data. The generative artificial intelligence (DALL-E 3) converts the photos and videos into 3D images and generates a 3D model of Hawaii to create a virtual reality space. The user accesses the 3D data and virtual reality space and shares an immersive experience with family and friends through VR goggles. During this process, the emotion engine recognizes the user's emotions and dynamically adjusts the settings of the virtual space based on that information.

[0738] Examples of prompt sentences that can be used include:

[0739] plain

[0740] Convert this family photo into a three-dimensional model.

[0741] Recreate Hawaii's tourist attractions as a three-dimensional model.

[0742] As a result, the present invention provides users with a more personalized, realistic, and immersive re-experience of their memories, and also serves as an effective means for deepening interactions with family and friends.

[0743] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0744] Step 1: Select and upload your data

[0745] Subject: User

[0746] Description: A user selects family photos and videos from their device and uploads them to the system. For example, the user selects photos and videos from a "Family Vacation" folder and enters the metadata, such as "Hawaii," as the location where the photos were taken.

[0747] Input: Photo data, video data, metadata (shooting location, date and time, etc.)

[0748] Output: Selected files and metadata are kept on your device

[0749] Specific behavior: Displays a file selection dialog and preserves the selected file and any entered metadata.

[0750] Step 2: Sending data

[0751] Subject: Terminal

[0752] Description: The terminal sends the selected file and the entered metadata to the server. The file is sent using an HTTP request.

[0753] Input: Selected files, metadata

[0754] Output: Notification of completion of transmission to the server

[0755] Specific behavior: Generates an HTTP POST request and sends the file and metadata to the server.

[0756] Step 3: Save the file data

[0757] Subject: Server

[0758] Description: The server receives the file data and metadata sent by the user and stores them in cloud storage. After storing them, it generates a file identifier and records it in the database.

[0759] Input: Submitted file data, metadata

[0760] Output: Saved to cloud storage and recorded in database

[0761] What it does: Upload a file to cloud storage, generate a file identifier, and then store the identifier and metadata in a database.

[0762] Step 4: Convert to 3D data

[0763] Subject: Server

[0764] Description: The server calls the generative AI model to analyze stored photo and video data. It inputs photo data along with a prompt to generate a three-dimensional model. Video data is similarly analyzed and converted into moving three-dimensional data.

[0765] Input: Saved photo data, video data, prompt (e.g., "Convert this family photo into a 3D model.")

[0766] Output: 3D data (3D models and moving models)

[0767] Specific operation: Call a generation AI (e.g., DALL-E 3), input a photo and a prompt, and generate a 3D model. Similarly, analyze video data to generate dynamic 3D data.

[0768] Step 5: Generate a 3D model of the location

[0769] Subject: Server

[0770] Description: The server generates a 3D model of the shooting location based on the metadata. For example, if the metadata includes "Hawaii," it generates a 3D model that recreates the scenery and tourist attractions of Hawaii.

[0771] Input: Metadata (location)

[0772] Output: 3D model of the location

[0773] What it does: Uses a generative AI model to generate a 3D model based on metadata (e.g., "Please recreate Hawaii's tourist attractions as a 3D, three-dimensional model.").

[0774] Step 6: Emotion Recognition

[0775] Subject: Emotion Engine

[0776] Description: When a user accesses a virtual reality space, the emotion engine recognizes the user's emotions. The emotion engine determines the user's current emotional state from their facial expressions, voice, and movements, and sends that data to the server.

[0777] Input: User's facial expression data, voice data, and movement data

[0778] Output: Emotional state data

[0779] Specific behavior: An emotion engine (e.g., Affectiva) analyzes the user's real-time data to determine their emotional state.

[0780] Step 7: Adjust based on sentiment data

[0781] Subject: Server

[0782] Description: The server analyzes data from the emotion engine and dynamically adjusts the settings of the virtual reality space according to the user's emotions. For example, if the user is in a "happy" emotional state, the virtual space will be set to bright colors and cheerful music.

[0783] Input: Emotional state data

[0784] Output: Calibrated virtual reality space configuration

[0785] Specific operation: The server analyzes the emotional data and changes the settings of the virtual reality space, such as color, music, and environment.

[0786] Step 8: Deliver the virtual reality experience

[0787] Subject: Server

[0788] Description: The server finally provides the user with a tailored virtual reality space. The user can wear VR goggles and experience family photos and videos in 3D, along with a 3D model of the shooting location, for an immersive experience.

[0789] Input: Calibrated virtual reality space

[0790] Output: User's virtual reality experience

[0791] Specific operation: The user puts on VR goggles, accesses a virtual reality space in real time, and experiences three-dimensional data and a coordinated environment.

[0792] (Application example 2)

[0793] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0794] Conventional virtual reality systems are unable to personalize the experience based on the user's emotions, and are limited to providing a uniform experience to users. In addition, in certain applications such as shopping experiences, personalization that dynamically reflects the user's interests and emotions is required, but this has been difficult to achieve.

[0795] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0796] In this invention, the server includes means for receiving photo data and video data, means for using a generation artificial intelligence to convert the photo data and video data into three-dimensional data, means for generating a virtual reality space from the three-dimensional data, means for providing an interface for a user to access and experience the virtual reality space, and means for using an emotion engine to recognize the user's emotions and dynamically adjust the virtual reality space. This enables the virtual reality space to be personalized based on the user's emotions, enabling individual suggestions and displays that reflect the user's interests and emotions, particularly in a shopping experience.

[0797] "Photo data" refers to data of still images taken by the user.

[0798] "Video data" refers to video data captured by the user.

[0799] "Three-dimensional data" is three-dimensional data generated based on photographic data and video data.

[0800] "Generative AI" is an AI that analyzes photo data and video data and generates three-dimensional data.

[0801] "Virtual reality space" is a virtual three-dimensional space generated based on three-dimensional data.

[0802] An "interface" is a device or software that allows a user to access and experience a virtual reality space.

[0803] The "emotion engine" is an engine that recognizes the user's emotions and dynamically adjusts the virtual reality space based on those emotions.

[0804] "Metadata" is additional information about photo data and video data, including the location and date of the photo.

[0805] The present invention relates to a technology for recognizing a user's emotions and dynamically adjusting a virtual reality space based on the emotions. Specific examples are given below.

[0806] First, the user selects and uploads family photos and videos from their device (e.g., smartphone or smart glasses). If necessary, the user can also enter metadata such as the shooting location and date and time. For example, the user can select photos and videos from the "Family Trip" folder and enter "Hawaii" as the shooting location. The device then sends the selected file data and the entered metadata to the server.

[0807] The server performs data processing and calculations using a series of hardware and software such as the following:

[0808] Generative AI model: The server uses a generative AI model to convert the received photo and video data into 3D data. Specifically, it analyzes 2D photos to generate a 3D model, and generates dynamic 3D data for video data.

[0809] Virtual reality environment tools: Based on the generated 3D data, a virtual reality space is constructed using virtual reality environment tools (e.g., Unity, Unreal Engine). In particular, a three-dimensional model of the shooting location is generated based on the metadata, and the model is incorporated into the virtual reality space.

[0810] Emotion engine: When a user accesses a virtual reality space, an emotion engine (e.g., Amazon Rekognition, Microsoft Emotion API) recognizes the user's emotional state from their facial expressions and voice. Based on this data, the server adjusts the VR environment in real time.

[0811] For example, if a user uploads photos from a "family trip" and specifies "Hawaii" as the location, the server receives and stores the data. The generative AI model then converts the photo and video data into 3D, generating a 3D model of Hawaii to create a virtual reality space. When the user then accesses the virtual reality space through smart glasses, the emotion engine recognizes the user's emotional state and, for example, provides a VR experience using bright colors and cheerful music if the user is having fun.

[0812] Below is an example of a prompt sentence:

[0813] Example prompt sentence:

[0814] "A user uploads a family photo. Use a generative AI model to convert the photo into 3D data and build a virtual reality space. Then, use an emotion engine to recognize the user's emotions and personalize the product displays in the virtual store based on those emotions."

[0815] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0816] Step 1:

[0817] Users select and upload family photos and videos from their own devices. The input includes photo data, video data, and metadata (such as the location and date of the photo). Based on this, the device processes the data by sending the file data and metadata to the server. The output is the photo data, video data, and metadata sent to the server.

[0818] Step 2:

[0819] The server receives and stores file data and metadata sent by users. The input is the photo data, video data, and metadata sent by users, and performs data operations to store it in storage. The output is the stored file data and metadata.

[0820] Step 3:

[0821] The server inputs the stored photo and video data into a generative AI model to generate 3D data. The input is the photo and video data stored in storage, and data processing is performed to convert the 2D data into 3D data. The output is 3D data.

[0822] Step 4:

[0823] The server generates a virtual reality space based on the 3D data. The input is the generated 3D data, which is then passed to a virtual reality environment tool for data calculations to construct a VR space. The output is the completed virtual reality space.

[0824] Step 5:

[0825] To recognize the user's emotions, the emotion engine analyzes their facial expressions and voice when they access the virtual reality space through a VR interface (e.g., smart glasses). The input is the user's facial expressions and voice data, which the emotion engine analyzes and processes. The output is the user's emotional state data.

[0826] Step 6:

[0827] The server receives emotional state data from the emotion engine and dynamically adjusts the virtual reality space based on that data. The input is the emotional state data from the emotion engine, and it performs data calculations to change the settings of the VR space based on that data. The output is a personalized virtual reality experience.

[0828] As a specific example, if a user uploads photos from a "family trip" and specifies "Hawaii" as the location, the data is sent from the device to the server in step 1, and the server stores them in step 2. In step 3, a generative AI model converts the photo and video data into 3D, and in step 4, a virtual reality space containing a 3D model of Hawaii is generated. Next, in step 5, an emotion engine analyzes the user's emotions, and in step 6, a personalized VR experience using bright colors and cheerful music is provided.

[0829] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0830] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0831] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0832] [Fourth embodiment]

[0833] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0834] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0835] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0836] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0837] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0838] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0839] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0840] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0841] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0842] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0843] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0844] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0845] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0846] This invention provides a system that converts photo data and video data into three-dimensional data and generates a virtual reality space based on that data. The system of the present invention is composed of three main elements: a user, a terminal, and a server.

[0847] First, the user selects and uploads family photos and videos from their device. The user can optionally enter metadata such as the shooting location and date and time. For example, the user can select photos and videos from the "Family Trip" folder and enter "Hawaii" as the shooting location.

[0848] Next, the device sends the file selected by the user and the metadata entered to the server. The server receives and stores the file data sent by the user. Based on the received file data, the server uses generative AI to convert the photo and video data into 3D data. For example, generative AI analyzes 2D photos and generates a 3D model of a family. It also does the same for video data, generating moving 3D data.

[0849] The server also generates a 3D model of the shooting location based on the metadata as needed. For example, if "Hawaii" is entered as metadata, a 3D model recreating the scenery and tourist spots of Hawaii is generated. A virtual reality space is then constructed based on this 3D model.

[0850] The server then saves the generated 3D data and virtual reality space in a user-accessible state and generates a preview link. The user can access the generated 3D data and virtual reality space and view it on their own device. For example, the user can click the preview link to view the 3D image or video in a browser.

[0851] Furthermore, users can use VR goggles to immerse themselves in the virtual reality space and relive their past experiences. They can also invite other family members and friends to the virtual reality space and relive shared memories in an immersive way. For example, a user can send an invitation link to their family members, allowing them to explore a virtual Hawaiian space together and share memories.

[0852] This system allows users to easily turn conventional flat photos and videos into three dimensions, allowing them to relive memories in a more immersive way. It also has the advantage of deepening interactions with family and friends, allowing them to share realistic experiences with family members who live far away.

[0853] For example, if a user selects photos and videos from a "Family Trip" folder and enters metadata about Hawaii, the system works as follows: The user first uploads the files from their device to the server, which receives and stores the data. The generative AI then creates a 3D model of Hawaii and constructs a virtual reality space. The user can then access the 3D data and virtual reality space and share the immersive experience with family and friends through VR goggles.

[0854] As described above, the present invention provides users with a means for reliving memories in a more realistic and convenient manner.

[0855] The processing flow will be explained below.

[0856] Step 1:

[0857] The user selects photos and videos from their device and presses the upload button. If necessary, the user can enter metadata such as the shooting location and date and time.

[0858] Step 2:

[0859] The terminal sends the selected file data and input metadata to the server, which receives the request and stores the file data and metadata.

[0860] Step 3:

[0861] The server sends the stored photo and video data to the generation AI, which analyzes the 2D data and generates a corresponding 3D model. Through this process, the generation AI outputs the three-dimensional data.

[0862] Step 4:

[0863] The server analyzes the metadata and generates a 3D model of the location where the photo was taken, if necessary. For example, if the metadata includes "Hawaii," it generates a 3D model of Hawaiian scenery and tourist attractions.

[0864] Step 5:

[0865] The server combines the generated 3D data with the virtual reality space and stores it as data for each user. It also generates a preview link so that the user can access this data.

[0866] Step 6:

[0867] Users click the preview link to view the 3D image or video. The device downloads the 3D data from the server and displays it in the browser.

[0868] Step 7:

[0869] Users put on VR goggles and log in to the virtual reality space, where they can enjoy an immersive experience with their family and friends.

[0870] Step 8:

[0871] The user invites other family members and friends to the virtual reality space, and the device sends an invitation link to the family and friends, allowing everyone to access the virtual reality space.

[0872] The above is the specific flow of the program's processing, which allows the user to re-experience memories in a three-dimensional and realistic way.

[0873] Example 1

[0874] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0875] Conventional photos and videos are two-dimensional, and the visual information they provide is limited, making it difficult for users to feel a sense of presence or a deep experience. Furthermore, the means for sharing those experiences with family and friends who live far away are also limited. This makes it difficult to recreate memories in a more realistic and immersive way, and to deepen shared experiences.

[0876] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0877] In this invention, the server includes means for receiving photo data and video data, means for using artificial intelligence to convert the photo data and video data into three-dimensional data, means for generating a virtual reality space based on the three-dimensional data and metadata, and means for providing an interface for users to access and experience the virtual reality space. This allows users to convert two-dimensional photos and videos into three-dimensional space, re-create memories in an immersive way, and share realistic experiences with family and friends who live far away.

[0878] "Photo data and video data" refers to digital files of still images and videos taken by users.

[0879] "Means for receiving" refers to the structure or method for transferring and storing digital files from a user's terminal to a server.

[0880] "Means using generative artificial intelligence" refers to a structure or method for converting two-dimensional photographs or videos into three-dimensional data using artificial intelligence techniques.

[0881] "Three-dimensional data" refers to a three-dimensional digital object created based on two-dimensional information from photographs and videos.

[0882] "Means for generating a virtual reality space" refers to a structure or method for integrating the generated three-dimensional data and creating a virtual three-dimensional space that can be experienced by a user.

[0883] "Means for providing an interface" refers to the operating screen or device that allows the user to access and experience the generated virtual reality space.

[0884] "Metadata" refers to additional information associated with a photo or video, such as the location and date it was taken.

[0885] A "prompt" is an instruction given to a generative AI model to perform a specific task.

[0886] "Means for generating 3D models" refers to a structure or method for creating a three-dimensional digital object using metadata and photographic and video data.

[0887] This invention provides a system that converts photo data and video data into three-dimensional data and generates a virtual reality space based on that data. The system of the present invention is composed of three main elements: a user, a terminal, and a server.

[0888] First, users can select photos and videos from their devices and upload them to the system. In addition, users can enter metadata such as the location and date of the photos. For example, a user can select photos and videos from a "Family Vacation" folder and enter "Hawaii" as the location.

[0889] The device sends the selected file and the entered metadata to the server, where it verifies that the photo data (e.g., JPEG file) or video data (e.g., MP4 file) is in the appropriate format and then transfers the data to the server.

[0890] The server stores the received photo and video data, as well as metadata. Using a generative AI model, the server converts the photo and video data into three-dimensional data. At this point, the server inputs a prompt into the generative AI model to generate the three-dimensional data. For example, the prompt could be, "This photo was taken on a family trip on a beach in Hawaii. Please generate a 3D model of the person in the photo and create a virtual reality space that recreates the Hawaiian background."

[0891] The generative AI model analyzes photo data to create a 3D model. Similarly, it analyzes video data frame by frame to generate a dynamic 3D model. This allows for the creation of 3D digital objects of family and friends.

[0892] Furthermore, the server can generate a 3D model of the shooting location based on the metadata. For example, if the metadata "Hawaii" is entered, a 3D model of Hawaii's scenery and tourist attractions will be generated. This will create a virtual reality space.

[0893] The server saves the generated 3D data and virtual reality space in a form accessible to the user and generates a preview link. By providing this link to the user, the user can check the 3D data and virtual reality space on their own device. The user can immerse themselves in the virtual reality space using VR goggles or similar devices and experience it within.

[0894] As a specific example, if a user selects photos and videos from a "Family Trip" folder and enters metadata about Hawaii, the system will operate as follows: The user uploads the files from their device to the server, which receives and stores the data. The generative AI then converts the photos and videos into 3D images, generating a 3D model of Hawaii and constructing a virtual reality space. The user can then access the 3D data and virtual reality space and share the immersive experience with family and friends through VR goggles.

[0895] This system allows users to easily create 3D images of traditional 2D photos and videos, and recreate them in a realistic way, deepening interactions with family and friends and allowing them to share realistic experiences with family members who live far away.

[0896] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0897] Step 1:

[0898] Users select photos and video files from their devices and upload them to the system. They can also enter metadata (such as the location and date of the photo). The input data is a JPEG photo file or an MP4 video file. The output is the selected file and the entered metadata.

[0899] Step 2:

[0900] The device sends the file selected by the user and the entered metadata to the server. The device first checks the file format to ensure it is in the appropriate format (e.g., JPEG or MP4). It then transfers this data to the server. The input is the file and metadata, and the output is a confirmation of successful transfer to the server.

[0901] Step 3:

[0902] The server receives and stores the photo data, video data, and metadata sent from the device. The storage destination is internal storage or cloud storage (e.g., S3 bucket). The input is the data sent from the device, and the output is the path to the saved file.

[0903] Step 4:

[0904] The server uses a generative AI model to convert photo and video data into 3D data. During this process, the server generates a prompt and inputs it into the generative AI. An example of a prompt is, "This photo was taken on a family trip on a beach in Hawaii. Please generate 3D models of the people in the photo and create a virtual reality space that recreates the Hawaiian background." The server passes this prompt to the AI ​​model and receives the resulting 3D data. The inputs are photo data, video data, and the prompt, and the output is 3D data.

[0905] Step 5:

[0906] The server generates a 3D model of the filming location based on the metadata. For example, if the metadata includes "Hawaii," the server generates a 3D model of Hawaii's scenery and tourist attractions. The server searches and selects this from an existing 3D library, or generates a new one as needed. The input is the metadata, and the output is a 3D model of the filming location.

[0907] Step 6:

[0908] The server integrates the generated 3D data with a 3D model of the shooting location to construct a virtual reality space. This virtual reality space is then saved as a single dataset so that it can be experienced by the user. The input is the 3D data and the 3D model of the shooting location, and the output is a dataset of the integrated virtual reality space.

[0909] Step 7:

[0910] The server generates an access link to the generated virtual reality space and provides it to the user. For example, the server generates a preview link and sends it to the user's email address. The input is a dataset of the virtual reality space, and the output is a preview link to the user.

[0911] Step 8:

[0912] Users click the provided preview link to view the 3D data and virtual reality space on their own devices. Furthermore, users can use devices such as VR goggles to immerse themselves in the virtual reality space. The input is the preview link, and the output is the experience of the virtual reality space.

[0913] Through these steps, the system is able to create three-dimensional images of the user's photos and videos, recreating them in a realistic way.

[0914] (Application example 1)

[0915] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0916] Conventional photo and video viewing is two-dimensional, making it difficult for users to experience a sense of realism. Furthermore, when converting a physical store into a virtual reality space, there is also the problem of it being cumbersome to check detailed product information. Therefore, there is a demand for a system that allows users to check detailed product information without actually visiting the store, providing an experience that makes them feel as if they are in the actual store.

[0917] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0918] In this invention, the server includes means for receiving photo data and video data, means for using artificial intelligence to convert the photo data and video data into three-dimensional data, means for generating a virtual reality space from the three-dimensional data, means for providing an interface for users to access and experience the virtual reality space, and means for providing an interactive function for users to check detailed product information in the virtual reality space. This allows users to easily check detailed product information in the virtual reality space without visiting a physical store, and to experience the product with a sense of realism as if they were in the physical store.

[0919] "Photo data" refers to still image data taken with electronic devices such as digital cameras and smartphones.

[0920] "Moving image data" refers to dynamic video data consisting of a series of still images.

[0921] "Three-dimensional data" is data that expresses the position and shape of an object or space in three-dimensional space.

[0922] "Generative AI" is AI that can generate new data based on input data.

[0923] A "virtual reality space" is a virtual three-dimensional space generated by a computer, allowing users to experience the feeling of being inside that space.

[0924] An "interface" is a means of communication between a user and a system.

[0925] An "interactive function" is a function that provides an immediate response to the user's operations on the system.

[0926] To practice the invention, the following procedures and system configurations are used.

[0927] First, users upload photos and video data of the interior of a physical store and products to the system using a device such as a smartphone or tablet. The user selects the photos and video data and enters metadata such as the shooting location and product information.

[0928] The device sends the selected file and the entered metadata to the server. The server receives and stores the photo and video data. Based on the received data, a generative AI model is used to convert the 2D photos and videos into 3D data. This generative AI model can use services such as Google Cloud AutoML Vision and Amazon Rekognition.

[0929] Based on the converted 3D data, the server uses a virtual reality space creation platform such as Unity 3D or Unreal Engine to generate a virtual reality space. The virtual reality space incorporates interactive functions such as the interior design of the actual store and detailed product information. This allows users to move freely through the virtual store using devices such as smartphones, head-mounted displays (HMDs), and smart glasses, and when they touch a product, detailed information about the product will pop up.

[0930] The generated virtual reality space is saved as a preview link online. Users can access this preview link and explore the store in the virtual reality space. This allows users to experience the store in a realistic way without visiting the actual store.

[0931] As a concrete example, consider a case where a store manager takes photos and product videos of the store and uploads them to the system. The metadata entered at this time could include, for example, a "new product section" or a "special sale area."

[0932] The generative AI model uses the following prompt:

[0933] 3D photo

[0934] "Generate a 3D model of a shop interior based on a 2D photo. Get the photo from the following link: [URL]"

[0935] 3D video

[0936] "Based on the uploaded video, please generate a 3D model of the product and 3D data that reproduces its movement. Please obtain the video data from the following link: [URL]"

[0937] Virtual reality store space

[0938] "Generate a virtual store environment based on the following metadata: Store name, location, product category, etc."

[0939] Finally, users can access a virtual reality space and experience an immersive store experience, allowing them to view detailed product information and make purchasing decisions without visiting a physical store.

[0940] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0941] Step 1:

[0942] Users upload photos and video data of store interiors and products from devices such as smartphones and tablets to the system. The input includes the photo and video data, as well as metadata such as the shooting location and product information, and the uploaded files are generated as output. The user operates the device to open the photo or video folder, select the file, and click the upload button.

[0943] Step 2:

[0944] The terminal sends the selected file and the entered metadata to the server. The terminal creates an HTTP request to send the file and sends it to the server. The input is the file and metadata selected by the user, and the output is the data sent to the server.

[0945] Step 3:

[0946] The server receives and stores photo and video data. It creates a directory structure to store the file data it receives in storage and writes the files. The data received from the device is input, and the saved file is generated as output.

[0947] Step 4:

[0948] Based on the received data, the server uses a generative AI model to convert photos and videos into 3D data. The server then sends API requests to Google Cloud AutoML Vision or Amazon Rekognition, analyzes the 2D data, and generates 3D data. The input is stored photo and video data, and the output is 3D data.

[0949] Step 5:

[0950] The server generates a virtual reality space based on the converted 3D data. The server reads the 3D data using the SDK of Unity 3D or Unreal Engine and performs the operation of constructing the virtual reality space programmatically. The generated 3D data is the input, and the virtual reality space is generated as the output.

[0951] Step 6:

[0952] The server implements interactive functions in the virtual reality space, allowing users to check detailed product information. The server sets up pop-up information and interaction events for each product's 3D model. The input is the virtual reality space and product information, and the output is a virtual reality space with interactive functions implemented.

[0953] Step 7:

[0954] The generated virtual reality space is saved as a link that can be previewed online. The server calls the link generation module to create a preview URL and saves it in the database. The input is the virtual reality space, and the output is a preview link.

[0955] Step 8:

[0956] Users can access the virtual reality space using the preview link and get a realistic store experience. Users access the link using a smartphone or head-mounted display and explore the virtual reality space. The preview link is the input, and access to the virtual reality space is possible as the output.

[0957] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0958] The present invention relates to a system that converts photo data and video data into three-dimensional data and generates a virtual reality space based on that data. One of the features of the present invention is that it can provide a more personalized virtual reality experience by combining it with an emotion engine that recognizes the user's emotions.

[0959] First, the user selects and uploads family photos and videos from their device. Users can optionally enter metadata such as the shooting location and date and time. For example, users can select photos and videos from the "Family Trip" folder and enter "Hawaii" as the shooting location.

[0960] Next, the device sends the selected file data and the entered metadata to the server. The server receives and stores the file data sent by the user. Based on the received file data, the server uses a generative AI to convert the photo and video data into 3D data. Specifically, the generative AI analyzes the 2D photos and generates a 3D model of the family. It also generates moving 3D data for the video data.

[0961] Furthermore, the server generates a 3D model of the shooting location based on the metadata as needed. For example, if "Hawaii" is entered as metadata, a 3D model recreating the scenery and tourist spots of Hawaii is generated. A virtual reality space is then constructed based on this 3D model.

[0962] This system adds a function in which the emotion engine recognizes the user's emotions and dynamically adjusts the virtual reality space based on that information. When a user accesses the virtual reality space, the emotion engine recognizes the user's current emotional state from their facial expressions, voice, and movements. For example, if the user is smiling, the system will determine that this is "fun."

[0963] The server analyzes the user's emotional data recognized by the emotion engine and provides the user with an appropriate virtual reality experience. For example, if the user is sad, the system will provide a virtual space with bright colors and cheerful music to help the user recall happy memories. On the other hand, if the user looks happy, the system will provide a virtual reality experience with a fast tempo that matches the user's emotions.

[0964] As a specific example, let's consider what happens when a user selects a photo from a "Family Trip" folder and enters metadata about Hawaii. The user first uploads the file from their device to the server, which receives and stores the data. The generative AI then converts the photos and videos into 3D images, generating a 3D model of Hawaii to create a virtual reality space. The user then accesses the 3D data and virtual reality space, sharing an immersive experience with family and friends through VR goggles. During this process, the emotion engine recognizes the user's emotions and dynamically adjusts the settings of the virtual space based on that information.

[0965] As a result, the present invention provides users with a more personalized, realistic, and immersive re-experience of their memories, and also serves as an effective means for deepening interactions with family and friends.

[0966] The processing flow will be explained below.

[0967] Step 1:

[0968] The user opens the "Family Trip" folder on their device, selects the photo IMG_001.jpg and the video VID_001.mp4, and presses the upload button. If necessary, the user can enter metadata such as "Hawaii" as the shooting location and "July 2023" as the date and time.

[0969] Step 2:

[0970] The terminal transmits the user's selected photo data and video data, as well as the input metadata, to the server, which receives the request and stores the file data and metadata.

[0971] Step 3:

[0972] The server sends the saved photo data IMG_001.jpg and video data VID_001.mp4 to the generation AI, which analyzes the photo data and video data and converts them into three-dimensional data.

[0973] Step 4:

[0974] The server analyzes the metadata and generates a 3D model of "Hawaii," the location specified for the shoot. The generated 3D data is combined with the 3D model of the shooting location to create a virtual reality space.

[0975] Step 5:

[0976] The server stores the generated 3D data and virtual reality space, and generates a preview link that the user can access. The user receives this preview link and checks it on their own device.

[0977] Step 6:

[0978] The user clicks the preview link to view the 3D images and videos. The device downloads the 3D data from the server and displays it in the browser.

[0979] Step 7:

[0980] The user puts on the VR goggles and logs in to experience the generated virtual reality space. In the virtual reality space, the user can enjoy an immersive experience.

[0981] Step 8:

[0982] The emotion engine recognizes the user's emotional state in real time from their facial expressions, voice, and actions. For example, if the user is smiling, the emotion engine will determine that they are "happy."

[0983] Step 9:

[0984] The server analyzes the emotion data sent from the emotion engine and dynamically adjusts the virtual reality space based on that information. For example, if the user is recognized as "happy," it will add bright colors and cheerful music to the virtual space.

[0985] Step 10:

[0986] The device sends an invitation link to allow the user to invite other family members or friends to the virtual reality space, and the family or friends who receive the link also log in to the virtual reality space and share the experience with the user.

[0987] Through these steps, users can transform traditional flat photos and videos into 3D images, enjoy personalized virtual reality experiences based on emotions, and deepen interactions with family and friends, even sharing realistic experiences with family members who live far away.

[0988] Example 2

[0989] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0990] While conventional systems could create 3D images of photo and video data, they were unable to dynamically adjust the virtual reality space to take into account the user's emotions. This made it difficult to provide a personalized experience for each user. Furthermore, their ability to generate 3D models of actual shooting locations was insufficient, limiting their ability to provide a more realistic and immersive experience.

[0991] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for receiving photo data and video data, a means for using a generation artificial intelligence to convert the photo data and video data into three-dimensional data, a means for generating a virtual reality space from the three-dimensional data, and a means for recognizing a user's emotions and dynamically adjusting the virtual reality space. This makes it possible to provide a personalized virtual reality experience according to the user's emotions, recreating the shooting location, and improving the sense of realism.

[0992] "Photo data" refers to still image data, which are digital images saved in formats such as JPEG or PNG.

[0993] "Video data" refers to data containing dynamic visual information consisting of a series of image frames, and is digital video saved in formats such as MP4 or AVI.

[0994] "Three-dimensional data" is data that contains three-dimensional information about an object or space, and is digital data that includes coordinate information on the X, Y, and Z axes.

[0995] A "virtual reality space" is a computer-generated three-dimensional environment that a user can experience visually and acoustically.

[0996] "Generative AI" is an artificial intelligence technology that mimics human intellectual processes and generates new information and models from given data.

[0997] An "interface" is a means for exchanging information between a user and a system, and includes GUIs (graphical user interfaces) and VR (virtual reality) goggles.

[0998] "Metadata" refers to additional information associated with a photo or video, such as the location, date, and time of the photo, and tags.

[0999] An "emotion recognition engine" is software or hardware that analyzes a user's facial expressions, voice, movements, etc., and determines their emotional state.

[1000] "Dynamic adjustment" refers to changing system settings and behavior in real time in response to changing conditions or environments.

[1001] "Storage means" refers to devices or methods for temporarily or permanently retaining data, including databases and cloud storage.

[1002] This invention is a system that converts photo and video data into three-dimensional data and generates a virtual reality space based on that data. In particular, by combining this with technology that recognizes the user's emotions and dynamically adjusts the virtual reality space, it is possible to provide a more personalized virtual reality experience.

[1003] A user can use their own device to select family photos and videos and upload them to the system. For example, the user selects photos and videos from a "Family Trip" folder and enters metadata such as "Hawaii" as the location where the photos were taken. The device then sends the selected files (photos and videos) and the entered metadata to the server using an HTTP request.

[1004] The server receives the file data and metadata sent by the user and stores them in cloud storage (e.g., Amazon S3). After saving, it generates a file identifier and records it in a database. The server then uses generative artificial intelligence (e.g., OpenAI's DALL-E 3) to convert the photo and video data into three-dimensional data. Specifically, the photo is input into the generative AI along with a prompt sentence, which generates a three-dimensional three-dimensional model. Video data is also analyzed in the same way and converted into moving three-dimensional data.

[1005] The server then generates a 3D model of the location based on the metadata. For example, if the metadata includes "Hawaii," the server uses a generative AI model to generate a 3D model of Hawaii's scenery and tourist attractions. This forms the framework for the virtual reality space.

[1006] When a user accesses a virtual reality space, an emotion engine (e.g., Affectiva) recognizes the user's emotions. The emotion engine determines the user's current emotional state from their facial expressions, voice, and movements, and sends this data to the server. The server analyzes the emotion recognition data and dynamically adjusts the colors and music of the virtual reality space according to the user's emotions. For example, if the user is in an emotional state of "fun," the virtual space will be set up with bright colors and cheerful music.

[1007] Finally, the server provides the user with a tailored virtual reality space. Using VR goggles, the user can experience three-dimensional family photos and videos, as well as a reconstructed location (e.g., Hawaii) in real time. The emotion engine continuously monitors the user's emotions and adjusts the virtual space accordingly.

[1008] As a specific example, let's consider the process when a user selects photos from a "Family Trip" folder and enters metadata for Hawaii. The user uploads photos and videos from their device, and the server receives and stores the data. The generative artificial intelligence (DALL-E 3) converts the photos and videos into 3D images and generates a 3D model of Hawaii to create a virtual reality space. The user accesses the 3D data and virtual reality space and shares an immersive experience with family and friends through VR goggles. During this process, the emotion engine recognizes the user's emotions and dynamically adjusts the settings of the virtual space based on that information.

[1009] Examples of prompt sentences that can be used include:

[1010] plain

[1011] Convert this family photo into a three-dimensional model.

[1012] Recreate Hawaii's tourist attractions as a three-dimensional model.

[1013] As a result, the present invention provides users with a more personalized, realistic, and immersive re-experience of their memories, and also serves as an effective means for deepening interactions with family and friends.

[1014] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1015] Step 1: Select and upload your data

[1016] Subject: User

[1017] Description: A user selects family photos and videos from their device and uploads them to the system. For example, the user selects photos and videos from a "Family Vacation" folder and enters the metadata, such as "Hawaii," as the location where the photos were taken.

[1018] Input: Photo data, video data, metadata (shooting location, date and time, etc.)

[1019] Output: Selected files and metadata are kept on your device

[1020] Specific behavior: Displays a file selection dialog and preserves the selected file and any entered metadata.

[1021] Step 2: Sending data

[1022] Subject: Terminal

[1023] Description: The terminal sends the selected file and the entered metadata to the server. The file is sent using an HTTP request.

[1024] Input: Selected files, metadata

[1025] Output: Notification of completion of transmission to the server

[1026] Specific behavior: Generates an HTTP POST request and sends the file and metadata to the server.

[1027] Step 3: Save the file data

[1028] Subject: Server

[1029] Description: The server receives the file data and metadata sent by the user and stores them in cloud storage. After storing them, it generates a file identifier and records it in the database.

[1030] Input: Submitted file data, metadata

[1031] Output: Saved to cloud storage and recorded in database

[1032] What it does: Upload a file to cloud storage, generate a file identifier, and then store the identifier and metadata in a database.

[1033] Step 4: Convert to 3D data

[1034] Subject: Server

[1035] Description: The server calls the generative AI model to analyze stored photo and video data. It inputs photo data along with a prompt to generate a three-dimensional model. Video data is similarly analyzed and converted into moving three-dimensional data.

[1036] Input: Saved photo data, video data, prompt (e.g., "Convert this family photo into a 3D model.")

[1037] Output: 3D data (3D models and moving models)

[1038] Specific operation: Call a generation AI (e.g., DALL-E 3), input a photo and a prompt, and generate a 3D model. Similarly, analyze video data to generate dynamic 3D data.

[1039] Step 5: Generate a 3D model of the location

[1040] Subject: Server

[1041] Description: The server generates a 3D model of the shooting location based on the metadata. For example, if the metadata includes "Hawaii," it generates a 3D model that recreates the scenery and tourist attractions of Hawaii.

[1042] Input: Metadata (location)

[1043] Output: 3D model of the location

[1044] What it does: Uses a generative AI model to generate a 3D model based on metadata (e.g., "Please recreate Hawaii's tourist attractions as a 3D, three-dimensional model.").

[1045] Step 6: Emotion Recognition

[1046] Subject: Emotion Engine

[1047] Description: When a user accesses a virtual reality space, the emotion engine recognizes the user's emotions. The emotion engine determines the user's current emotional state from their facial expressions, voice, and movements, and sends that data to the server.

[1048] Input: User's facial expression data, voice data, and movement data

[1049] Output: Emotional state data

[1050] Specific behavior: An emotion engine (e.g., Affectiva) analyzes the user's real-time data to determine their emotional state.

[1051] Step 7: Adjust based on sentiment data

[1052] Subject: Server

[1053] Description: The server analyzes data from the emotion engine and dynamically adjusts the settings of the virtual reality space according to the user's emotions. For example, if the user is in a "happy" emotional state, the virtual space will be set to bright colors and cheerful music.

[1054] Input: Emotional state data

[1055] Output: Calibrated virtual reality space configuration

[1056] Specific operation: The server analyzes the emotional data and changes the settings of the virtual reality space, such as color, music, and environment.

[1057] Step 8: Deliver the virtual reality experience

[1058] Subject: Server

[1059] Description: The server finally provides the user with a tailored virtual reality space. The user can wear VR goggles and experience family photos and videos in 3D, along with a 3D model of the shooting location, for an immersive experience.

[1060] Input: Calibrated virtual reality space

[1061] Output: User's virtual reality experience

[1062] Specific operation: The user puts on VR goggles, accesses a virtual reality space in real time, and experiences three-dimensional data and a coordinated environment.

[1063] (Application example 2)

[1064] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1065] Conventional virtual reality systems are unable to personalize the experience based on the user's emotions, and are limited to providing a uniform experience to users. In addition, in certain applications such as shopping experiences, personalization that dynamically reflects the user's interests and emotions is required, but this has been difficult to achieve.

[1066] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1067] In this invention, the server includes means for receiving photo data and video data, means for using a generation artificial intelligence to convert the photo data and video data into three-dimensional data, means for generating a virtual reality space from the three-dimensional data, means for providing an interface for a user to access and experience the virtual reality space, and means for using an emotion engine to recognize the user's emotions and dynamically adjust the virtual reality space. This enables the virtual reality space to be personalized based on the user's emotions, enabling individual suggestions and displays that reflect the user's interests and emotions, particularly in a shopping experience.

[1068] "Photo data" refers to data of still images taken by the user.

[1069] "Video data" refers to video data captured by the user.

[1070] "Three-dimensional data" is three-dimensional data generated based on photographic data and video data.

[1071] "Generative AI" is an AI that analyzes photo data and video data and generates three-dimensional data.

[1072] "Virtual reality space" is a virtual three-dimensional space generated based on three-dimensional data.

[1073] An "interface" is a device or software that allows a user to access and experience a virtual reality space.

[1074] The "emotion engine" is an engine that recognizes the user's emotions and dynamically adjusts the virtual reality space based on those emotions.

[1075] "Metadata" is additional information about photo data and video data, including the location and date of the photo.

[1076] The present invention relates to a technology for recognizing a user's emotions and dynamically adjusting a virtual reality space based on the emotions. Specific examples are given below.

[1077] First, the user selects and uploads family photos and videos from their device (e.g., smartphone or smart glasses). If necessary, the user can also enter metadata such as the shooting location and date and time. For example, the user can select photos and videos from the "Family Trip" folder and enter "Hawaii" as the shooting location. The device then sends the selected file data and the entered metadata to the server.

[1078] The server performs data processing and calculations using a series of hardware and software such as the following:

[1079] Generative AI model: The server uses a generative AI model to convert the received photo and video data into 3D data. Specifically, it analyzes 2D photos to generate a 3D model, and generates dynamic 3D data for video data.

[1080] Virtual reality environment tools: Based on the generated 3D data, a virtual reality space is constructed using virtual reality environment tools (e.g., Unity, Unreal Engine). In particular, a three-dimensional model of the shooting location is generated based on the metadata, and the model is incorporated into the virtual reality space.

[1081] Emotion engine: When a user accesses a virtual reality space, an emotion engine (e.g., Amazon Rekognition, Microsoft Emotion API) recognizes the user's emotional state from their facial expressions and voice. Based on this data, the server adjusts the VR environment in real time.

[1082] For example, if a user uploads photos from a "family trip" and specifies "Hawaii" as the location, the server receives and stores the data. The generative AI model then converts the photo and video data into 3D, generating a 3D model of Hawaii to create a virtual reality space. When the user then accesses the virtual reality space through smart glasses, the emotion engine recognizes the user's emotional state and, for example, provides a VR experience using bright colors and cheerful music if the user is having fun.

[1083] Below is an example of a prompt sentence:

[1084] Example prompt sentence:

[1085] "A user uploads a family photo. Use a generative AI model to convert the photo into 3D data and build a virtual reality space. Then, use an emotion engine to recognize the user's emotions and personalize the product displays in the virtual store based on those emotions."

[1086] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1087] Step 1:

[1088] Users select and upload family photos and videos from their own devices. The input includes photo data, video data, and metadata (such as the location and date of the photo). Based on this, the device processes the data by sending the file data and metadata to the server. The output is the photo data, video data, and metadata sent to the server.

[1089] Step 2:

[1090] The server receives and stores file data and metadata sent by users. The input is the photo data, video data, and metadata sent by users, and performs data operations to store it in storage. The output is the stored file data and metadata.

[1091] Step 3:

[1092] The server inputs the stored photo and video data into a generative AI model to generate 3D data. The input is the photo and video data stored in storage, and data processing is performed to convert the 2D data into 3D data. The output is 3D data.

[1093] Step 4:

[1094] The server generates a virtual reality space based on the 3D data. The input is the generated 3D data, which is then passed to a virtual reality environment tool for data calculations to construct a VR space. The output is the completed virtual reality space.

[1095] Step 5:

[1096] To recognize the user's emotions, the emotion engine analyzes their facial expressions and voice when they access the virtual reality space through a VR interface (e.g., smart glasses). The input is the user's facial expressions and voice data, which the emotion engine analyzes and processes. The output is the user's emotional state data.

[1097] Step 6:

[1098] The server receives emotional state data from the emotion engine and dynamically adjusts the virtual reality space based on that data. The input is the emotional state data from the emotion engine, and it performs data calculations to change the settings of the VR space based on that data. The output is a personalized virtual reality experience.

[1099] As a specific example, if a user uploads photos from a "family trip" and specifies "Hawaii" as the location, the data is sent from the device to the server in step 1, and the server stores them in step 2. In step 3, a generative AI model converts the photo and video data into 3D, and in step 4, a virtual reality space containing a 3D model of Hawaii is generated. Next, in step 5, an emotion engine analyzes the user's emotions, and in step 6, a personalized VR experience using bright colors and cheerful music is provided.

[1100] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1101] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1102] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1103] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1104] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1105] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1106] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1107] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1108] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1109] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1110] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1111] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1112] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1113] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1114] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1115] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1116] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1117] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1118] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1119] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1120] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1121] The following is further disclosed regarding the above embodiment.

[1122] (Claim 1)

[1123] means for receiving photo data and video data;

[1124] A means for using artificial intelligence to convert photographic data and video data into three-dimensional data;

[1125] A means for generating a virtual reality space from three-dimensional data;

[1126] a means for providing an interface for a user to access and experience the virtual reality space;

[1127] A system including:

[1128] (Claim 2)

[1129] 10. The system of claim 1, further comprising means for generating a three-dimensional model of the location based on the photographic data and the video data.

[1130] (Claim 3)

[1131] 10. The system of claim 1, further comprising means for analyzing user-uploaded metadata and generating a three-dimensional model based on that information.

[1132] "Example 1"

[1133] (Claim 1)

[1134] means for receiving photo data and video data;

[1135] A means for using artificial intelligence to convert photographic data and video data into three-dimensional data;

[1136] means for generating a virtual reality space based on the three-dimensional data and metadata;

[1137] a means for providing an interface for a user to access and experience the virtual reality space;

[1138] A system including:

[1139] (Claim 2)

[1140] 10. The system of claim 1, further comprising means for generating a three-dimensional model of the location based on the photographic data and the video data.

[1141] (Claim 3)

[1142] 10. The system of claim 1, further comprising means for analyzing metadata uploaded by a user and generating prompt sentences corresponding to the metadata to generate a three-dimensional model based on the information.

[1143] "Application Example 1"

[1144] (Claim 1)

[1145] means for receiving photo data and video data;

[1146] A means for using artificial intelligence to convert photographic data and video data into three-dimensional data;

[1147] A means for generating a virtual reality space from three-dimensional data;

[1148] a means for providing an interface for a user to access and experience the virtual reality space;

[1149] A means for providing an interactive function for users to check detailed product information in a virtual reality space;

[1150] A system including:

[1151] (Claim 2)

[1152] 10. The system of claim 1, further comprising means for generating a three-dimensional model of the location based on the photographic data and the video data.

[1153] (Claim 3)

[1154] 10. The system of claim 1, further comprising means for analyzing user-uploaded metadata and generating a three-dimensional model based on that information.

[1155] "Example 2: Combining Emotion Engines"

[1156] (Claim 1)

[1157] means for receiving photo data and video data;

[1158] A means for using artificial intelligence to convert photographic data and video data into three-dimensional data;

[1159] A means for generating a virtual reality space from three-dimensional data;

[1160] a means for providing an interface for a user to access and experience the virtual reality space;

[1161] means for recognizing a user's emotion and dynamically adjusting the virtual reality space;

[1162] A system including:

[1163] (Claim 2)

[1164] 10. The system of claim 1, further comprising means for generating a three-dimensional model of the location based on the photographic data and the video data.

[1165] (Claim 3)

[1166] 10. The system of claim 1, further comprising means for analyzing user-uploaded metadata and generating a three-dimensional model based on that information.

[1167] (Claim 4)

[1168] 10. The system of claim 1, further comprising means for storing the transmitted file data and metadata.

[1169] (Claim 5)

[1170] 10. The system of claim 1, further comprising means for recognizing emotions from a user's facial expression, voice, and actions using an emotion recognition engine.

[1171] (Claim 6)

[1172] 10. The system of claim 1, further comprising means for dynamically adjusting colors and music of the virtual reality space based on the recognized emotion data.

[1173] "Application example 2 when combining emotion engines"

[1174] (Claim 1)

[1175] means for receiving photo data and video data;

[1176] A means for using artificial intelligence to convert photographic data and video data into three-dimensional data;

[1177] A means for generating a virtual reality space from three-dimensional data;

[1178] a means for providing an interface for a user to access and experience the virtual reality space;

[1179] a means for using an emotion engine that recognizes a user's emotion and dynamically adjusts the virtual reality space;

[1180] A system including:

[1181] (Claim 2)

[1182] 10. The system of claim 1, further comprising means for generating a three-dimensional model of the location based on the photographic data and the video data.

[1183] (Claim 3)

[1184] 10. The system of claim 1, further comprising means for analyzing user-uploaded metadata and generating a three-dimensional model based on that information. [Explanation of symbols]

[1185] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving photo data and video data; A means for using artificial intelligence to convert photographic data and video data into three-dimensional data; A means for generating a virtual reality space from three-dimensional data; a means for providing an interface for a user to access and experience the virtual reality space; A system including:

2. The system of claim 1 further comprising means for generating a three-dimensional model of the location based on the photographic data and the video data.

3. The system of claim 1 , further comprising means for analyzing user-uploaded metadata and generating a three-dimensional model based on that information.

Images