System

Abstract

A system is provided.SOLUTION: A system comprising: means for processing a request received from a user and generating an input datum for item generation; means for automatically generating an item based on the input datum using a generative AI model; means for providing the generated item datum to the user and performing item generation again in response to a check and correction request; and means for registering and arranging a finally determined item datum in a Metaverse space.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Creating and enriching content in the metaverse space requires specialized knowledge and skills, making it difficult for general users. This can delay the development of the metaverse space as a whole. Furthermore, there is a lack of mechanisms for users to quickly generate content that reflects their own needs. The purpose of this invention is to solve these problems and enable users to efficiently create high-quality items without specialized knowledge, thereby enriching the metaverse space. [Means for solving the problem]

[0005] The present invention is a system including a means for processing requests received from a user and generating input data for item generation, a means for automatically generating an item based on the input data using a generative AI model, a means for providing the generated item data to the user and generating the item again in response to confirmation and correction requests, and a means for registering and arranging the finalized item data in a metaverse space. The system further includes a format conversion means for converting the user's request into input data, allowing the user to easily input requests, and a process in which the generative AI model generates 3D models and textures, enabling the generation of detailed, high-quality items.

[0006] A "user" is an entity that creates items and inputs requests in the metaverse space.

[0007] A "request" is the instructions or desired information a user provides to a generative AI model to generate a specific item.

[0008] "Input data" is data that processes user requests and is used as input to a generative AI model.

[0009] A "generative AI model" is an artificial intelligence algorithm or system that automatically generates items based on input data.

[0010] "Item" refers to an object or content used within the Metaverse space, which may include 3D models and textures.

[0011] "Item data" refers to detailed information and data files about items generated by generative AI models.

[0012] The "format conversion means" is a function that converts requests input by a user into an appropriate format as input data for the generative AI model. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] The system of the present invention automatically generates items in the Metaverse space based on user requests. This system includes a series of processes: the user inputs the request using a terminal, the server processes the request data and uses a generation AI model, the server provides the generated results and reflects correction instructions, and the final item data is registered and placed in the Metaverse space.

[0035] First, a user uses a terminal to input a request for creating an item. For example, if a user wants to create a "blue modern sofa," the user inputs detailed instructions such as "sofa," "blue," and "modern" into the interface. The terminal converts these requests into an appropriate format (e.g., JSON format) and sends them to the server.

[0036] Next, the server receives the request data sent by the user. The server analyzes this data and processes it as input data for the generative AI model. The generative AI model generates a 3D model and texture of the item based on this input data. The generated item data is temporarily stored on the server.

[0037] The generated item data is sent from the server to the device for confirmation by the user. The user can check the generated results provided on the device and input correction instructions as necessary. For example, if the user wants to change the color of the sofa to a "lighter blue," they can send that instruction again from the device to the server. The server will then use the generative AI model again based on the new instructions to generate the corrected item. This allows the user to make repeated corrections until they achieve the desired result.

[0038] Finally, if the user is satisfied with the created item, the server registers this item data in the metaverse space. The registered item can then be placed and used by the user in the metaverse space. The user can then adjust the position and orientation of the item in their metaverse space to complete the room decoration.

[0039] Specific examples

[0040] The specific flow when a user creates a "blue modern sofa" is shown below.

[0041] 1. Request Input

[0042] The user types "sofa," "blue," and "modern" into the device interface.

[0043] The terminal converts this into JSON format and sends it to the server.

[0044] 2. Processing requirements data and using generative AI models

[0045] The server receives and analyzes the request data.

[0046] The generative AI model is given the inputs "sofa," "blue," and "modern," and generates a 3D model and texture of a blue modern sofa.

[0047] 3. Providing and Modifying Generated Results

[0048] The server transmits the generated sofa data to the terminal.

[0049] The user checks the result and inputs a correction request such as "make it a little brighter blue."

[0050] The server inputs the modification request back into the generative AI model and generates the modified sofa.

[0051] 4. Registration and placement in the metaverse space

[0052] Finally, the server registers the sofa that the user is satisfied with in the metaverse space.

[0053] The user places the sofa in the metaverse space and completes the decoration.

[0054] The system allows users to easily generate and distribute high-quality content without specialized knowledge.

[0055] The processing flow will be explained below.

[0056] Step 1:

[0057] The user enters a request

[0058] The user uses the device interface to input a request for the item they want to create. For example, to specify a "blue modern sofa," they input "sofa," "blue," and "modern." The device then converts this request into JSON-formatted data.

[0059] Step 2:

[0060] The device sends the requested data to the server

[0061] The terminal sends the converted request data to the server as an HTTP request. The server analyzes the received request data and converts it into a format that can be handled internally.

[0062] Step 3:

[0063] The server inputs data into the generative AI model

[0064] The server passes the analyzed request data as input to the generative AI model, which then generates a 3D model and texture for the requested item based on the input data.

[0065] Step 4:

[0066] Generative AI model generates items

[0067] The generative AI model automatically generates a 3D model and texture of a blue modern sofa based on the specifications of "sofa," "blue," and "modern." The generated item data is then returned to the server.

[0068] Step 5:

[0069] The server returns the generated results to the terminal

[0070] The server organizes the item data returned by the generative AI model and sends it back to the device, which then displays the generated results to the user.

[0071] Step 6:

[0072] The user checks the generated results and inputs correction requests as necessary.

[0073] The user checks the generated item on the device. For example, if the color is different from what they expected, they input a correction request into the device, such as "make it a lighter blue." The device then reformats this into JSON and sends it to the server.

[0074] Step 7:

[0075] The server inputs the correction request into the generated AI model again.

[0076] The server receives the user's modification request and inputs the data back into the generative AI model, which then generates a new item based on the modification request.

[0077] Step 8:

[0078] The generative AI model generates the modified item.

[0079] The generative AI model generates a modified 3D model and texture based on the new input data and returns it to the server, which then sends the modified item data back to the device.

[0080] Step 9:

[0081] The user confirms the item and confirms it

[0082] The user reviews the results again and repeats steps 6 through 8 as necessary until satisfied, finally finalizing the items they are satisfied with.

[0083] Step 10:

[0084] The server registers the item data in the metaverse space.

[0085] The server registers the confirmed item data in the metaverse space database. The item is reflected in the user's metaverse space.

[0086] Step 11:

[0087] Users place items in the metaverse space

[0088] The user logs in to the metaverse space using a terminal and places the generated items wherever they like. The user adjusts the position and orientation of the items to complete the final layout.

[0089] Example 1

[0090] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0091] In the Metaverse, it has been difficult for users to easily generate high-quality items and place them as desired. Conventional systems require advanced expertise and complex operations, placing a heavy burden on many users. In addition, there is also the problem that if a generated item does not completely meet the user's requirements, it is not easy to regenerate or modify it.

[0092] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0093] In this invention, the server includes means for processing requests received from users and generating input data for generating items, means for converting the user's request into JSON format using a terminal and transmitting the JSON data to the server, means for analyzing the JSON data received by the server and formatting it as input data for the generative AI model, means for automatically generating items based on the input data using the generative AI model, means for temporarily storing the generated item data on the server, means for providing the generated item data to the user and generating items again in response to the user's confirmation and correction requests, means for registering and arranging item data that the user is satisfied with in the metaverse space, and means for adjusting the position and orientation of registered items in the metaverse space. This enables users without specialized knowledge to easily generate high-quality items, make corrections until they meet their requirements, and finally place them in the metaverse space.

[0094] "User" refers to an individual or organization that uses this system to generate and place items.

[0095] "Terminal" refers to a computer or mobile device used by a User to register input data and view generated items.

[0096] "Server" refers to a computer system that receives user requests, analyzes the request data, generates items using a generative AI model, and registers the item data in the Metaverse space.

[0097] "Means for generating input data" refers to the function of converting a request received from a user into the data format (e.g., JSON format) required for item generation.

[0098] "JSON format" is an abbreviation for JavaScript (registered trademark) Object Notation, and refers to a lightweight data exchange format for expressing data in a format that is easy for humans to read and machines to analyze.

[0099] "Generative AI model" refers to an artificial intelligence model that automatically generates items such as 3D models and textures based on input data.

[0100] "Item Data" refers to digital data, including 3D models and textures, generated by a generative AI model.

[0101] The "metaverse space" is a virtual digital world, an online environment where users can place and interact with registered items.

[0102] "Means for adjusting position and orientation" refers to the function that allows users to change the position and orientation of registered items in the metaverse space and optimize their placement.

[0103] "Means for generating an item again in response to confirmation and correction requests" refers to the function that, when the user confirms the generated results provided and inputs a correction request, regenerates the item using the generation AI model again based on that request.

[0104] The system of the present invention automatically generates items in the Metaverse space based on user requests. This system includes a series of processes: the user inputs the request using a terminal, the server processes the request data and uses a generation AI model, the server provides the generated results and reflects correction instructions, and the final item data is registered and placed in the Metaverse space.

[0105] First, a user uses a terminal to input a request for creating an item. For example, if a user wants to create a "blue modern sofa," the user inputs detailed instructions such as "sofa," "blue," and "modern" into the interface. The terminal converts these requests into an appropriate format (e.g., JSON format) and sends them to the server.

[0106] Next, the server receives the request data sent by the user. The server analyzes this data and formats it as input data for the generative AI model. The generative AI model generates a 3D model and texture of the item based on this input data. The generated item data is temporarily stored on the server.

[0107] The generated item data is sent from the server to the device for confirmation by the user. The user can check the generated results provided on the device and input correction instructions as necessary. For example, if the user wants to change the color of the sofa to a "lighter blue," they can send that instruction again from the device to the server. The server will then use the generative AI model again based on the new instructions to generate the corrected item. This allows the user to make repeated corrections until they achieve the desired result.

[0108] Finally, if the user is satisfied with the created item, the server registers this item data in the metaverse space. The registered item can then be placed and used by the user in the metaverse space. The user can then adjust the position and orientation of the item in their metaverse space to complete the room decoration.

[0109] Hardware and software used

[0110] Server: A high-performance server (e.g., a server equipped with an NVIDIA GPU)

[0111] Generative AI model: Examples of models used are OpenAI's (registered trademark) DALL-E and Google's (registered trademark) DeepMind.

[0112] Data format: JSON

[0113] Specific examples

[0114] The specific flow when a user creates a "blue modern sofa" is shown below.

[0115] 1. Request Input

[0116] The user types "sofa," "blue," and "modern" into the device interface.

[0117] The terminal converts this into JSON format and sends it to the server.

[0118] 2. Processing requirements data and using generative AI models

[0119] The server receives and analyzes the request data.

[0120] The generative AI model is given the inputs "sofa," "blue," and "modern," and generates a 3D model and texture of a blue modern sofa.

[0121] 3. Providing and Modifying Generated Results

[0122] The server transmits the generated sofa data to the terminal.

[0123] The user checks the result and inputs a correction request such as "make it a little brighter blue."

[0124] The server inputs the modification request back into the generative AI model and generates the modified sofa.

[0125] 4. Registration and placement in the metaverse space

[0126] Finally, the server registers the sofa that the user is satisfied with in the metaverse space.

[0127] The user places the sofa in the metaverse space and completes the decoration.

[0128] Prompt Sentence Examples

[0129] Here are some example input prompts for a generative AI model:

[0130] Example prompt

[0131] "Sofa" "Blue" "Modern"

[0132] The system allows users to easily generate and distribute high-quality content without requiring specialized knowledge.

[0133] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0134] Step 1:

[0135] The user inputs a request for item creation into the terminal interface. The user inputs a specific request, such as "sofa," "blue," and "modern," into the terminal. This causes the terminal to prepare the data for step 2.

[0136] Input: User request (e.g. "sofa", "blue", "modern")

[0137] Output: Request to convert input data to JSON format

[0138] Specific behavior:

[0139] The user enters a keyword into the input field on the device.

[0140] The terminal receives input data.

[0141] Step 2:

[0142] The terminal converts the user's request into JSON format. The converted JSON data is sent to the server. At this stage, the data format is unified and formatted so that it can be passed to the server.

[0143] Input: User request data

[0144] Output: Data converted to JSON format

[0145] Specific behavior:

[0146] The terminal converts the input data into JSON format.

[0147] The terminal sends the converted JSON data to the server as an HTTP request.

[0148] Step 3:

[0149] The server parses the received JSON data and formats it as input data for the generative AI model. The JSON data is parsed and converted into a prompt that the AI ​​model can understand.

[0150] Input: User request data in JSON format

[0151] Output: Input data to the generative AI model

[0152] Specific behavior:

[0153] The server deserializes and parses the JSON data.

[0154] The server formats the data as required.

[0155] Step 4:

[0156] The server inputs the formatted data into a generative AI model to generate a 3D model and texture for the item. The generative AI model generates the item based on the input data, and the data is temporarily stored on the server.

[0157] Input: Formatted input data

[0158] Output: 3D model and texture data of the generated item

[0159] Specific behavior:

[0160] The server sends the formatted data to the generative AI model.

[0161] Generative AI model generates 3D models and textures.

[0162] The server temporarily stores the generated data.

[0163] Step 5:

[0164] The server sends the generated item data to the terminal and receives confirmation from the user. The user confirms the provided generation results on the terminal and proceeds to the next step.

[0165] Input: Generated item data

[0166] Output: Generated data provided to the user

[0167] Specific behavior:

[0168] The server sends the generated data to the terminal as an HTTP response.

[0169] The device displays the provided data.

[0170] Step 6:

[0171] The user checks the generated results and inputs correction instructions as needed. For example, they may request a correction such as "make it a lighter blue." The device then converts this new instruction back into JSON format and sends it to the server.

[0172] Input: User correction instructions

[0173] Output: Request to convert correction instructions to JSON format

[0174] Specific behavior:

[0175] The user enters correction instructions into the interface.

[0176] The device converts the correction instructions into JSON format and sends them to the server.

[0177] Step 7:

[0178] The server receives the modification instructions, inputs them into the generation AI model again, and regenerates the item. The item is modified based on the new instructions.

[0179] Input: Correction instruction data

[0180] Output: 3D model and texture data of the modified item

[0181] Specific behavior:

[0182] The server analyzes the correction instructions and re-inputs them into the generative AI model.

[0183] A generative AI model generates modified items.

[0184] The server saves the modified data.

[0185] Step 8:

[0186] If the user is satisfied with the final generated item, the server registers this item data in the metaverse space.

[0187] Input: Satisfied item data

[0188] Output: Registration data in the metaverse space

[0189] Specific behavior:

[0190] The user makes a final confirmation and indicates satisfaction.

[0191] The server registers the item data in the metaverse space.

[0192] Step 9:

[0193] Users can complete the decoration of a room by placing registered items in the metaverse space and adjusting their position and orientation.

[0194] Input: Registered item data

[0195] Output: Position and orientation of placed items

[0196] Specific behavior:

[0197] A user logs into the metaverse space.

[0198] The user arranges the items and adjusts their position and orientation.

[0199] Through this series of steps, users can easily create high-quality items without any specialized knowledge, repeatedly modify them until they meet their requirements, and finally place them in the Metaverse space.

[0200] (Application example 1)

[0201] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0202] Conventional methods for building metaverses and virtual stores require users to have specialized design knowledge, making them difficult for average users. Customizing and arranging items also requires a lot of time and effort, making efficient design difficult. Therefore, there was a demand for a system that would allow users to easily design high-quality virtual stores.

[0203] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0204] In this invention, the server includes means for processing requests received from users and generating input data for generating items, means for automatically generating items based on the input data using a generative AI model, means for providing the generated item data to the user and generating items again in response to confirmation and correction requests, means for registering and arranging the finalized item data in the metaverse space, and means for providing a visual interface for arranging and customizing the generated items in a virtual store. This enables users, even without specialized knowledge, to easily and efficiently design and customize high-quality virtual stores.

[0205] "Means for processing requests received from users and generating input data for item generation" is a function that analyzes requests entered by users and converts them into data in a format suitable for item generation.

[0206] "Means for automatically generating items based on the input data using a generative AI model" refers to a function that passes input data to an AI model and automatically generates items such as 3D models and textures based on that data.

[0207] "Means for providing the generated item data to the user and generating the item again in response to confirmation and correction requests" is a function that shows the generated item data to the user and, if necessary, generates the item again by reflecting correction requests made by the user.

[0208] The "means for registering and arranging the finalized item data in the metaverse space" is a function for registering the item data that the user has finally confirmed in the metaverse space and arranging it.

[0209] "Means for providing a visual interface for arranging and customizing items generated in a virtual store" is a function that provides a user with an interface that allows them to visually arrange and customize items generated in a virtual store.

[0210] The "format conversion means" is a means for converting a user request into input data.

[0211] The "process of generating a 3D model and texture" refers to the procedure by which an AI model generates a three-dimensional shape and its surface material based on a user's request.

[0212] A system for implementing the present invention automatically generates items for a virtual store based on user requests. This system includes a series of processes: a user inputs the request using a terminal, a server processes the request data and uses a generation AI model, a server provides the generated results and reflects correction instructions, and finally, the system registers and places the final item data in the virtual store.

[0213] First, a user uses a terminal to input a request for creating an item. For example, if a user wants to create a "wooden shelf" or a "retro lamp," the user inputs these requests through the interface. The terminal converts these requests into an appropriate format (e.g., JSON format) and sends them to the server.

[0214] Next, the server receives the request data sent by the user. The server analyzes this data and processes it as input data for the generative AI model. The generative AI model used by the server is generally composed of AI technologies such as GPT-4 (registered trademark) and Stable Diffusion. The generative AI model generates a 3D model and texture of the item based on the input data. The generated item data is temporarily stored on the server.

[0215] The generated item data is sent from the server to the device for confirmation by the user. The user can check the generated results on the device and input correction instructions as needed. For example, a request for a "sofa with a lighter blue color" is possible. The server then uses the generative AI model again based on the new instructions to generate the corrected item. This allows the user to repeatedly make corrections until they achieve the desired result.

[0216] Finally, if the user is satisfied with the generated items, the server registers the item data in the virtual store space. The registered items become available for the user to place and use in the virtual store. The user adjusts the position and orientation of the items in their virtual store to complete the design of the store interior.

[0217] To give a specific example, when a user wants to generate a "wooden shelf," the following steps are taken: First, the user enters "wooden shelf" into the device interface. The device then converts this into JSON format and sends it to the server. The server receives the request data, passes "wooden shelf" as input to the generative AI model, and generates a 3D model and texture. The generated shelf data is sent to the device for confirmation by the user. If necessary, a correction request can be made, such as "make it a lighter color," and the corrected data is then generated again.

[0218] The above system configuration enables users to easily and efficiently design and customize high-quality virtual stores, even without specialized knowledge.

[0219] Example prompt sentences to use:

[0220] "Create a modern, high-quality blue sofa."

[0221] "Please change it to a lighter blue."

[0222] "Generate a wooden shelf."

[0223] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0224] Step 1:

[0225] The user inputs a request to create an item through the terminal interface. The user inputs a detailed prompt such as "wooden shelf" or "retro lamp." The input prompt is converted into an appropriate format (e.g., JSON format) by the terminal. Here, the user's request is the input, and the converted formatted data is the output.

[0226] Step 2:

[0227] The terminal transmits the format-converted data to the server. Specifically, the terminal uploads the converted data to the server via the network. Here, the format-converted data is the input, and the requested data received by the server is the output.

[0228] Step 3:

[0229] The server analyzes the received request data and creates input data for the generative AI model. Specifically, the server analyzes the JSON format data and formats it in a format that the generative AI model can understand. Here, the received request data is received as input, and the input data for the generative AI model is obtained as output.

[0230] Step 4:

[0231] The server uses a generative AI model to generate a 3D model and texture for the item. Specifically, the server passes input data to a generative AI model (such as GPT-4 or Stable Diffusion) and generates the item based on that data. Here, the input to the generative AI model is the input, and the generated 3D model and texture data are obtained as the output.

[0232] Step 5:

[0233] The server sends the generated 3D model and texture data to the terminal and provides it to the user. Specifically, the server sends the generated results to the terminal via the network. Here, the generated 3D model and texture data are the input, and the generated results displayed on the terminal are obtained as the output.

[0234] Step 6:

[0235] The user uses the terminal to check the generated results and input correction requests as necessary. The user may input a correction request such as "make the color a little brighter" into the terminal. Here, the generated item data and the user's correction request are input, and the correction request is sent from the terminal to the server as output.

[0236] Step 7:

[0237] The server receives the modification request and generates the modified item using the generative AI model again. Specifically, the server analyzes the modification request and passes it to the generative AI model again to generate a new 3D model and texture. Here, the modification request is the input and the modified 3D model and texture data are obtained as the output.

[0238] Step 8:

[0239] Finally, if the user is satisfied with the generated item, the server registers this item data in the virtual store space. Specifically, the server saves the item data in storage so that it can be placed in the user's virtual store. Here, the input is the finalized item data, and the output is the item registered in the virtual store.

[0240] Step 9:

[0241] Users use their devices to arrange and customize items in a virtual store. Specifically, they adjust the position and orientation of items using a visual interface. Here, the input is the items displayed in the virtual store, and the output is the final store design.

[0242] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0243] The system of the present invention not only automatically generates items in the metaverse space based on user requests, but also recognizes the user's emotions and suggests or modifies items according to those emotions. This system involves a series of processes: the user inputs the request using a terminal, the server processes the request data and uses a generation AI model and emotion engine, the server provides the generated results and reflects modification instructions, and the final item data is registered and placed in the metaverse space.

[0244] First, a user uses a terminal to input a request for creating an item. For example, if a user wants to create a "blue modern sofa," the user inputs detailed instructions such as "sofa," "blue," and "modern" into the interface. The terminal converts these requests into an appropriate format (e.g., JSON format) and sends them to the server.

[0245] Next, the server receives the request data sent by the user. The server analyzes this data and processes it as input data for the generative AI model and emotion engine. The generative AI model generates a 3D model and texture for the item based on this input data. The emotion engine also recognizes the user's emotion and adjusts the parameters of the generative AI model to generate an item that best suits the user's emotion. The generated item data is temporarily stored on the server.

[0246] The generated item data is sent from the server to the device for user confirmation. The user can check the generated results on the device and input correction instructions as necessary. For example, if the color of a sofa differs from what the user's emotions were considered, the emotion engine analyzes the user's emotions from their facial expressions, voice, and text input and suggests appropriate corrections. If the user accepts the suggestions, the device reformats the correction instructions in JSON format and sends them to the server.

[0247] The server uses the generative AI model based on new instructions and the user's emotional data to generate a revised item. This allows the user to repeatedly make revisions until they achieve the desired result. Finally, if the user is satisfied with the generated item, the server registers this item data in the metaverse space. The registered item can then be placed and used by the user within the metaverse space. The user can then adjust the position and orientation of the item within their metaverse space to complete the decoration of their room.

[0248] Specific examples

[0249] The specific flow when a user creates a "blue modern sofa" is shown below.

[0250] 1. Request Input

[0251] The user types "sofa," "blue," and "modern" into the device interface.

[0252] The terminal converts this into JSON format and sends it to the server.

[0253] 2. Processing of request data and use of generation AI models and emotion engines

[0254] The server receives and analyzes the request data.

[0255] The generative AI model is given the inputs "sofa," "blue," and "modern," and generates a 3D model and texture of a blue modern sofa.

[0256] The emotion engine analyzes the user's facial expressions and voice and adjusts the parameters of the generative AI model.

[0257] 3. Providing and Modifying Generated Results

[0258] The server transmits the generated sofa data to the terminal.

[0259] The user checks the result and inputs a correction request such as "make it a little brighter blue."

[0260] The emotion engine analyzes the user's emotions and suggests the most suitable colors.

[0261] The server inputs the modification request back into the generative AI model and generates the modified sofa.

[0262] 4. Registration and placement in the metaverse space

[0263] Finally, the server registers the sofa that the user is satisfied with in the metaverse space.

[0264] The user places the sofa in the metaverse space and completes the decoration.

[0265] This system allows users to easily generate and arrange high-quality content without specialized knowledge, and also provides optimal items that take into account the user's emotions.

[0266] The processing flow will be explained below.

[0267] Step 1:

[0268] The user enters a request

[0269] The user uses the device interface to input a request for the item they want to create. For example, to specify a "blue modern sofa," they input "sofa," "blue," and "modern." The device then converts this request into JSON-formatted data.

[0270] Step 2:

[0271] The device sends the requested data to the server

[0272] The terminal sends the converted request data to the server as an HTTP request. The server analyzes the received request data and converts it into a format that can be handled internally.

[0273] Step 3:

[0274] The server inputs data into the generated AI model and emotion engine

[0275] The server passes the analyzed request data to the generative AI model and emotion engine as input data. The generative AI model generates a 3D model and texture of the requested item based on the input data. Meanwhile, the emotion engine collects and analyzes data such as the user's facial expressions, voice, and text input to analyze the user's emotions.

[0276] Step 4:

[0277] Generative AI model and emotion engine generate items and emotion data

[0278] The generative AI model automatically generates a 3D model and texture of a blue modern sofa based on the specifications of "sofa," "blue," and "modern." Meanwhile, the emotion engine analyzes the user's input data and real-time emotion data to suggest the optimal items to satisfy the user. The generated item data and emotion data are returned to the server.

[0279] Step 5:

[0280] The server returns the generated results to the terminal

[0281] The server organizes the generated item data and the analysis results of the emotion engine and sends them back to the device, which then displays the generated results and suggestions to the user.

[0282] Step 6:

[0283] The user can review the generated results and suggestions and enter correction requests if necessary.

[0284] The user checks the generated items and the emotion engine's suggestions on the device. For example, if the color is different from what they expected, the user can input a correction request into the device, such as "make it a lighter blue." The device then converts this into JSON format and sends it to the server.

[0285] Step 7:

[0286] The server generates a correction request again and inputs it into the AI ​​model and emotion engine.

[0287] The server receives the user's modification request and again inputs the data into the generative AI model and emotion engine. The generative AI model generates a modified 3D model and texture based on the new input data, and the emotion engine generates new emotion data and suggestions. The generated data is returned to the server and sent back to the device.

[0288] Step 8:

[0289] The user confirms the item and confirms it

[0290] The user reviews the results again and repeats steps 6 and 7 as necessary until satisfied, finally finalizing the satisfactory items.

[0291] Step 9:

[0292] The server registers the item data in the metaverse space.

[0293] The server registers the confirmed item data in the metaverse space database. The item is reflected in the user's metaverse space.

[0294] Step 10:

[0295] Users place items in the metaverse space

[0296] The user logs in to the metaverse space using a terminal and places the generated items wherever they like. The user adjusts the position and orientation of the items to complete the final layout.

[0297] Example 2

[0298] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0299] In conventional systems, it was difficult for users to generate specific items while taking their emotions into consideration and providing optimal items. Furthermore, reflecting requests for revisions to generated items required manual work and multiple trial and error processes, which was inefficient. There was also a need for a system that could centrally manage these processes and be intuitive for users to operate.

[0300] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0301] In this invention, the server includes means for processing requests received from users and generating input data for generation, means for automatically generating items based on the input data using a generative model, means for providing the generated item data to the user and generating the item again in response to a request for confirmation and correction, means for recognizing the user's emotions and adjusting the parameters of the generative model in accordance with the emotions, and means for registering and arranging the finalized item data in a virtual space. This allows users to efficiently generate and modify optimal items that reflect their emotions and arrange them in a virtual space.

[0302] "User" refers to a person who utilizes a terminal to input requests into the system and to review and modify generated items.

[0303] "Server" refers to a computer system that receives request data from a user and generates and modifies items using generative models and emotion recognition engines.

[0304] A "terminal" refers to a device operated by a user to input requests and communicate with a server. Specifically, this applies to smartphones and computers.

[0305] "Request data" refers to data obtained by converting detailed information about an item that a user inputs through a terminal into a different format.

[0306] A "generative model" refers to an artificial intelligence model that automatically generates 3D models and textures based on required data.

[0307] An "emotion recognition engine" refers to a software system that analyzes a user's facial expressions and voice to identify the user's emotions.

[0308] "Virtual space" refers to a three-dimensional computer-generated environment constructed using digital technology that users can manipulate and experience as if it were a physical space through an interface.

[0309] "Input data" refers to data used by the generative model based on user request data.

[0310] The system of the present invention automatically generates items in a virtual space based on user requests, recognizes the user's emotions, and suggests or modifies items according to the emotions. This system includes a series of processes: request input, request data processing, use of a generation AI model and an emotion recognition engine, provision of the generated results and reflection of modification requests, and final registration and placement of the item data in the virtual space.

[0311] First, a user uses a terminal to input a request for creating an item. For example, if a user wants to create a "blue modern sofa," the user inputs detailed instructions such as "sofa," "blue," and "modern" into the interface. The terminal converts these requests into an appropriate format (e.g., JSON format) and sends them to the server.

[0312] The server receives and analyzes the request data sent by the user. The analyzed data is passed to the generative AI model and emotion recognition engine. The generative AI model generates a 3D model and texture of the item based on this input data. Meanwhile, the emotion recognition engine recognizes the user's emotion and adjusts the parameters of the generative AI model. For example, if the user is smiling, the emotion recognition engine instructs the generated item to use bright colors. The generated item data is temporarily stored on the server.

[0313] The generated item data is sent from the server to the device for user confirmation. The user can review the generated results on the device and input correction requests as needed. For example, if the color of the sofa is darker than expected, the user can input a correction request such as "make it a lighter blue." The emotion recognition engine analyzes the user's emotions and the new request and suggests the optimal correction. The server then uses the generative AI model again based on the new instructions to generate the corrected item.

[0314] Finally, if the user is satisfied with the generated item, the server registers the item data in the virtual space. The registered item can then be placed and used by the user in the virtual space. The user can then adjust the position and orientation of the item in their virtual space to complete the room decoration.

[0315] Specific examples

[0316] A specific example prompt for a user to generate a "blue modern sofa" would be:

[0317] "Generate a blue modern sofa."

[0318] "Please suggest a blue sofa that reflects the emotion you read from the user's facial expression."

[0319] This system allows users to easily create and arrange high-quality items without specialized knowledge, and also provides optimal items that take into account the user's emotions.

[0320] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0321] Step 1: User enters item creation request

[0322] The user inputs details of the item they want to create (e.g., "sofa," "blue," "modern") into the device interface. The device receives this as input data and converts it into JSON format using a format conversion tool. The input is the user's instructions, and the output is JSON format data.

[0323] Step 2: The terminal converts the requested data into a different format and sends it

[0324] The terminal converts the item creation request entered by the user into JSON format and then sends this data to the server. Here, the input is the detailed request data from the user, and the output is the JSON data sent to the server.

[0325] Step 3: Parsing and processing the request data by the server

[0326] The server receives the JSON data sent from the device and analyzes it. The analyzed content includes the item type, color, style, etc. The analyzed data is passed as input to the generative AI model and emotion recognition engine. The input is JSON data, and the output is the analyzed item information.

[0327] Step 4: Generate items using a generative AI model

[0328] The server passes the analyzed item information to the generative AI model, which generates a 3D model and texture according to the specified conditions (e.g., "blue modern sofa"). The generated data is temporarily stored on the server. The input is the analyzed item information, and the output is the generated 3D model and texture.

[0329] Step 5: Recognize and adjust the user's emotions using the emotion recognition engine

[0330] The server uses an emotion recognition engine to recognize the user's emotions. This involves analyzing the user's facial expressions and voice using the device's camera and microphone. The input is the user's facial expression and voice data, and the output is the recognized emotional information. This emotional information is used to adjust the parameters of the generative AI model and reflect it in the output.

[0331] Step 6: Server saves and sends the generated results

[0332] The server temporarily stores the generated and adjusted item data and then transmits it to the user's device. The input is the adjusted item data, and the output is the data sent to the device.

[0333] Step 7: User checks the generated results and requests corrections

[0334] The user checks the generated item on the terminal interface and inputs correction requests as needed. For example, the user may request "make it a lighter blue." The input is the generated item data, and the output is the user's correction requests.

[0335] Step 8: Emotion Recognition Engine Suggests Corrections

[0336] The emotion recognition engine proposes optimal corrections based on the user's new request and the results of facial and voice analysis. The input is the user's new request and emotional data, and the output is the optimal correction proposal.

[0337] Step 9: Server creates and resubmits the corrected item

[0338] The server uses the generative AI model again based on the user's correction request to generate a corrected item. This corrected item is then resent to the user's device. The input is the correction instruction and emotion data, and the output is the corrected item data.

[0339] Step 10: Register and place items in the metaverse space

[0340] The server registers the items that the user is finally satisfied with in the virtual space. After registration is complete, the user can freely place items in the virtual space and customize the environment. The input is the finalized item data, and the output is the items that have been registered in the virtual space and can now be placed.

[0341] (Application example 2)

[0342] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0343] Previously, when creating items in a virtual store or metaverse space, users had to manually input and edit data using a limited interface, which was extremely time-consuming and laborious. Furthermore, if the created item did not match the user's emotions or preferences, the user would have to re-edit it, resulting in a poor user experience. The present invention aims to solve these problems and provide a system that allows users to create and edit items more comfortably and efficiently.

[0344] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing the user's emotions in real time and adjusting the parameters of the generation AI model based on the emotion data, means for providing the generated item data to the user and generating items again in response to confirmation and correction requests, and means for registering and arranging the finalized item data in the virtual space. This allows the user to easily generate and modify optimal items based on their emotions and preferences, and efficiently arrange them.

[0345] "Means for recognizing a user's emotions in real time and adjusting the parameters of a generative AI model based on the emotion data" refers to a system that has the ability to recognize emotions in real time not only from data entered by the user but also from the user's facial expressions, voice, etc., and dynamically adjust the operating parameters of a generative AI model according to those emotions.

[0346] A "generative AI model" is an artificial intelligence model that automatically generates specific items based on input data, including processes that generate 3D models and textures.

[0347] The "emotion engine" analyzes the user's emotions and suggests or modifies items based on that emotion data, thereby providing the optimal items that the user desires.

[0348] "Virtual space" refers to a virtual area where users can engage in various activities within a digital space, such as a metaverse environment or a virtual store provided via the Internet.

[0349] The "means for generating an item again in response to a confirmation and correction request" is a system function that, when a user confirms a generated item and requests correction as necessary, generates the item again based on that request.

[0350] "Means for registering and arranging finalized item data in a virtual space" is a function that allows a user to register items that have been confirmed to their satisfaction in a virtual space and arrange them freely within that virtual space.

[0351] The "format conversion means for converting a user's request into input data" is a means having the function of appropriately formatting a request input by a user and converting it into a form that is easily understood by the system.

[0352] The system according to the present invention automatically generates items in a virtual space based on requests from a user, and further recognizes the user's emotions and suggests or modifies items according to the emotions.

[0353] System Program

[0354] Basic configuration

[0355] User terminal: A device used by users to input requests and check generated items. This can be a smart glass or a smartphone.

[0356] Server: The central device that processes request data and generates items using generative AI models and emotion engines.

[0357] Generative AI model: An artificial intelligence model that generates items based on user requests.

[0358] Emotion engine: An engine that analyzes user emotion data and adjusts the parameters of the generative AI model based on that data.

[0359] Explanation of program processing

[0360] The server first receives a request entered by the user via their device. This request is converted into a data format such as JSON. The server then inputs the converted data into a generative AI model to generate a 3D model and texture of the item based on the request. The emotion engine also recognizes emotional data from the user's facial expressions, voice, and text input, and adjusts the parameters of the generative AI model.

[0361] The generated item is immediately sent to the user's terminal, where the user can confirm it. If the user requires any modifications, the request is sent again to the server. This allows the optimal item to be generated according to the user's feelings. Modifications continue until the user is satisfied, and the finalized item is registered in the virtual space.

[0362] Specific examples

[0363] Suppose a user accesses a virtual store using smart glasses and enters a request by voice, such as "a blue modern sofa." The keywords "sofa," "blue," and "modern" are transmitted to the server through the interface of the smart glasses.

[0364] The server receives this and generates a 3D model and texture of a "blue modern sofa." At the same time, the emotion engine detects a "relaxed atmosphere" from the user's smile and adjusts the color of the sofa slightly brighter accordingly. The generated result is presented to the user, and if the user requests a correction such as "a darker blue," the item is generated again using the same process.

[0365] Example prompt sentence:

[0366] "Create a modern blue sofa that has a relaxing atmosphere."

[0367] This system allows users to easily create items that best suit their emotions and preferences, and efficiently place them in the virtual space.

[0368] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0369] Step 1:

[0370] A user uses a device (such as smart glasses or a smartphone) to input requests such as "sofa," "blue," and "modern" into the user interface. This input request data is converted into a data format such as JSON and sent to the server.

[0371] Input: User request (e.g. "sofa", "blue", "modern")

[0372] Output: Formatted request data (e.g., JSON format)

[0373] Specific operation:

[0374] When a user inputs something using voice or gestures, the device analyzes it and converts it into text data, which it then converts into JSON format and sends to the server via the network.

[0375] Step 2:

[0376] The server receives and analyzes the request data sent from the device. The analyzed data is input into the generative AI model to generate a 3D model and texture of a "blue modern sofa." The generated temporal data is also adjusted by the emotion engine based on the user's emotional data.

[0377] Input: Formatted request data

[0378] Output: Generated item data (3D models and textures)

[0379] Specific operation:

[0380] The server analyzes the received JSON data and passes the keywords "sofa," "blue," and "modern" to the generative AI model. The generative AI model creates a 3D model and texture based on the data. At the same time, the emotion engine analyzes the user's emotion data and adjusts the parameters of the generative AI model to generate the optimal output.

[0381] Step 3:

[0382] The generated item data is temporarily stored and sent to the user's terminal. The user can check the generated item on the terminal and, if not satisfied, input a correction request. This correction request is sent back to the server.

[0383] Input: Generated item data

[0384] Output: Correction requests from the user

[0385] Specific operation:

[0386] The item data is sent from the server to the device, where the user confirms it. If the user wishes to make any changes, they can enter their request by voice or text, such as "make it a lighter blue." The device then converts this request back into JSON format and sends it to the server.

[0387] Step 4:

[0388] The server receives the modification request from the user again and generates the item again using the generative AI model and emotion engine, thereby providing an item optimized for the user's emotions.

[0389] Input: User request for correction (e.g., "Make it a lighter blue")

[0390] Output: Regenerated item data

[0391] Specific operation:

[0392] The server receives the correction request, inputs it back into the generative AI model as data, and uses the emotion engine to readjust it based on the user's emotions. The generative AI model then creates new 3D models and textures and sends the data to the user's device.

[0393] Step 5:

[0394] Finally, once an item that satisfies the user is created, the server registers and places the item data in the virtual space. The user can then adjust the position and orientation of the item within the virtual space to complete the decoration of the space.

[0395] Input: Finalized item data

[0396] Output: Items registered in the virtual space

[0397] Specific operation:

[0398] The server receives the item data that the user is ultimately satisfied with and registers it in the specified location within the virtual space. The user can then freely adjust the position and orientation of the items within the virtual space via their device to complete the decoration of the space.

[0399] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0400] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0401] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0402] [Second embodiment]

[0403] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0404] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0405] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0406] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0407] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0408] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0409] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0410] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0411] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0412] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0413] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0414] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0415] The system of the present invention automatically generates items in the Metaverse space based on user requests. This system includes a series of processes: the user inputs the request using a terminal, the server processes the request data and uses a generation AI model, the server provides the generated results and reflects correction instructions, and the final item data is registered and placed in the Metaverse space.

[0416] First, a user uses a terminal to input a request for creating an item. For example, if a user wants to create a "blue modern sofa," the user inputs detailed instructions such as "sofa," "blue," and "modern" into the interface. The terminal converts these requests into an appropriate format (e.g., JSON format) and sends them to the server.

[0417] Next, the server receives the request data sent by the user. The server analyzes this data and processes it as input data for the generative AI model. The generative AI model generates a 3D model and texture of the item based on this input data. The generated item data is temporarily stored on the server.

[0418] The generated item data is sent from the server to the device for confirmation by the user. The user can check the generated results provided on the device and input correction instructions as necessary. For example, if the user wants to change the color of the sofa to a "lighter blue," they can send that instruction again from the device to the server. The server will then use the generative AI model again based on the new instructions to generate the corrected item. This allows the user to make repeated corrections until they achieve the desired result.

[0419] Finally, if the user is satisfied with the created item, the server registers this item data in the metaverse space. The registered item can then be placed and used by the user in the metaverse space. The user can then adjust the position and orientation of the item in their metaverse space to complete the room decoration.

[0420] Specific examples

[0421] The specific flow when a user creates a "blue modern sofa" is shown below.

[0422] 1. Request Input

[0423] The user types "sofa," "blue," and "modern" into the device interface.

[0424] The terminal converts this into JSON format and sends it to the server.

[0425] 2. Processing requirements data and using generative AI models

[0426] The server receives and analyzes the request data.

[0427] The generative AI model is given the inputs "sofa," "blue," and "modern," and generates a 3D model and texture of a blue modern sofa.

[0428] 3. Providing and Modifying Generated Results

[0429] The server transmits the generated sofa data to the terminal.

[0430] The user checks the result and inputs a correction request such as "make it a little brighter blue."

[0431] The server inputs the modification request back into the generative AI model and generates the modified sofa.

[0432] 4. Registration and placement in the metaverse space

[0433] Finally, the server registers the sofa that the user is satisfied with in the metaverse space.

[0434] The user places the sofa in the metaverse space and completes the decoration.

[0435] The system allows users to easily generate and distribute high-quality content without specialized knowledge.

[0436] The processing flow will be explained below.

[0437] Step 1:

[0438] The user enters a request

[0439] The user uses the device interface to input a request for the item they want to create. For example, to specify a "blue modern sofa," they input "sofa," "blue," and "modern." The device then converts this request into JSON-formatted data.

[0440] Step 2:

[0441] The device sends the requested data to the server

[0442] The terminal sends the converted request data to the server as an HTTP request. The server analyzes the received request data and converts it into a format that can be handled internally.

[0443] Step 3:

[0444] The server inputs data into the generative AI model

[0445] The server passes the analyzed request data as input to the generative AI model, which then generates a 3D model and texture for the requested item based on the input data.

[0446] Step 4:

[0447] Generative AI model generates items

[0448] The generative AI model automatically generates a 3D model and texture of a blue modern sofa based on the specifications of "sofa," "blue," and "modern." The generated item data is then returned to the server.

[0449] Step 5:

[0450] The server returns the generated results to the terminal

[0451] The server organizes the item data returned by the generative AI model and sends it back to the device, which then displays the generated results to the user.

[0452] Step 6:

[0453] The user checks the generated results and inputs correction requests as necessary.

[0454] The user checks the generated item on the device. For example, if the color is different from what they expected, they input a correction request into the device, such as "make it a lighter blue." The device then reformats this into JSON and sends it to the server.

[0455] Step 7:

[0456] The server inputs the correction request into the generated AI model again.

[0457] The server receives the user's modification request and inputs the data back into the generative AI model, which then generates a new item based on the modification request.

[0458] Step 8:

[0459] The generative AI model generates the modified item.

[0460] The generative AI model generates a modified 3D model and texture based on the new input data and returns it to the server, which then sends the modified item data back to the device.

[0461] Step 9:

[0462] The user confirms the item and confirms it

[0463] The user reviews the results again and repeats steps 6 through 8 as necessary until satisfied, finally finalizing the items they are satisfied with.

[0464] Step 10:

[0465] The server registers the item data in the metaverse space.

[0466] The server registers the confirmed item data in the metaverse space database. The item is reflected in the user's metaverse space.

[0467] Step 11:

[0468] Users place items in the metaverse space

[0469] The user logs in to the metaverse space using a terminal and places the generated items wherever they like. The user adjusts the position and orientation of the items to complete the final layout.

[0470] Example 1

[0471] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0472] In the Metaverse, it has been difficult for users to easily generate high-quality items and place them as desired. Conventional systems require advanced expertise and complex operations, placing a heavy burden on many users. In addition, there is also the problem that if a generated item does not completely meet the user's requirements, it is not easy to regenerate or modify it.

[0473] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0474] In this invention, the server includes means for processing requests received from users and generating input data for generating items, means for converting the user's request into JSON format using a terminal and transmitting the JSON data to the server, means for analyzing the JSON data received by the server and formatting it as input data for the generative AI model, means for automatically generating items based on the input data using the generative AI model, means for temporarily storing the generated item data on the server, means for providing the generated item data to the user and generating items again in response to the user's confirmation and correction requests, means for registering and arranging item data that the user is satisfied with in the metaverse space, and means for adjusting the position and orientation of registered items in the metaverse space. This enables users without specialized knowledge to easily generate high-quality items, make corrections until they meet their requirements, and finally place them in the metaverse space.

[0475] "User" refers to an individual or organization that uses this system to generate and place items.

[0476] "Terminal" refers to a computer or mobile device used by a User to register input data and view generated items.

[0477] "Server" refers to a computer system that receives user requests, analyzes the request data, generates items using a generative AI model, and registers the item data in the Metaverse space.

[0478] "Means for generating input data" refers to the function of converting a request received from a user into the data format (e.g., JSON format) required for item generation.

[0479] "JSON format" is an abbreviation for JavaScript Object Notation, and refers to a lightweight data exchange format for expressing data in a format that is easy for humans to read and machines to parse.

[0480] "Generative AI model" refers to an artificial intelligence model that automatically generates items such as 3D models and textures based on input data.

[0481] "Item Data" refers to digital data, including 3D models and textures, generated by a generative AI model.

[0482] The "metaverse space" is a virtual digital world, an online environment where users can place and interact with registered items.

[0483] "Means for adjusting position and orientation" refers to the function that allows users to change the position and orientation of registered items in the metaverse space and optimize their placement.

[0484] "Means for generating an item again in response to confirmation and correction requests" refers to the function that, when the user confirms the generated results provided and inputs a correction request, regenerates the item using the generation AI model again based on that request.

[0485] The system of the present invention automatically generates items in the Metaverse space based on user requests. This system includes a series of processes: the user inputs the request using a terminal, the server processes the request data and uses a generation AI model, the server provides the generated results and reflects correction instructions, and the final item data is registered and placed in the Metaverse space.

[0486] First, a user uses a terminal to input a request for creating an item. For example, if a user wants to create a "blue modern sofa," the user inputs detailed instructions such as "sofa," "blue," and "modern" into the interface. The terminal converts these requests into an appropriate format (e.g., JSON format) and sends them to the server.

[0487] Next, the server receives the request data sent by the user. The server analyzes this data and formats it as input data for the generative AI model. The generative AI model generates a 3D model and texture of the item based on this input data. The generated item data is temporarily stored on the server.

[0488] The generated item data is sent from the server to the device for confirmation by the user. The user can check the generated results provided on the device and input correction instructions as necessary. For example, if the user wants to change the color of the sofa to a "lighter blue," they can send that instruction again from the device to the server. The server will then use the generative AI model again based on the new instructions to generate the corrected item. This allows the user to make repeated corrections until they achieve the desired result.

[0489] Finally, if the user is satisfied with the created item, the server registers this item data in the metaverse space. The registered item can then be placed and used by the user in the metaverse space. The user can then adjust the position and orientation of the item in their metaverse space to complete the room decoration.

[0490] Hardware and software used

[0491] Server: A high-performance server (e.g., a server equipped with an NVIDIA GPU)

[0492] Generative AI model: Examples of models include OpenAI's DALL-E and Google's DeepMind

[0493] Data format: JSON

[0494] Specific examples

[0495] The specific flow when a user creates a "blue modern sofa" is shown below.

[0496] 1. Request Input

[0497] The user types "sofa," "blue," and "modern" into the device interface.

[0498] The terminal converts this into JSON format and sends it to the server.

[0499] 2. Processing requirements data and using generative AI models

[0500] The server receives and analyzes the request data.

[0501] The generative AI model is given the inputs "sofa," "blue," and "modern," and generates a 3D model and texture of a blue modern sofa.

[0502] 3. Providing and Modifying Generated Results

[0503] The server transmits the generated sofa data to the terminal.

[0504] The user checks the result and inputs a correction request such as "make it a little brighter blue."

[0505] The server inputs the modification request back into the generative AI model and generates the modified sofa.

[0506] 4. Registration and placement in the metaverse space

[0507] Finally, the server registers the sofa that the user is satisfied with in the metaverse space.

[0508] The user places the sofa in the metaverse space and completes the decoration.

[0509] Prompt Sentence Examples

[0510] Here are some example input prompts for a generative AI model:

[0511] Example prompt

[0512] "Sofa" "Blue" "Modern"

[0513] The system allows users to easily generate and distribute high-quality content without requiring specialized knowledge.

[0514] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0515] Step 1:

[0516] The user inputs a request for item creation into the terminal interface. The user inputs a specific request, such as "sofa," "blue," and "modern," into the terminal. This causes the terminal to prepare the data for step 2.

[0517] Input: User request (e.g. "sofa", "blue", "modern")

[0518] Output: Request to convert input data to JSON format

[0519] Specific behavior:

[0520] The user enters a keyword into the input field on the device.

[0521] The terminal receives input data.

[0522] Step 2:

[0523] The terminal converts the user's request into JSON format. The converted JSON data is sent to the server. At this stage, the data format is unified and formatted so that it can be passed to the server.

[0524] Input: User request data

[0525] Output: Data converted to JSON format

[0526] Specific behavior:

[0527] The terminal converts the input data into JSON format.

[0528] The terminal sends the converted JSON data to the server as an HTTP request.

[0529] Step 3:

[0530] The server parses the received JSON data and formats it as input data for the generative AI model. The JSON data is parsed and converted into a prompt that the AI ​​model can understand.

[0531] Input: User request data in JSON format

[0532] Output: Input data to the generative AI model

[0533] Specific behavior:

[0534] The server deserializes and parses the JSON data.

[0535] The server formats the data as required.

[0536] Step 4:

[0537] The server inputs the formatted data into a generative AI model to generate a 3D model and texture for the item. The generative AI model generates the item based on the input data, and the data is temporarily stored on the server.

[0538] Input: Formatted input data

[0539] Output: 3D model and texture data of the generated item

[0540] Specific behavior:

[0541] The server sends the formatted data to the generative AI model.

[0542] Generative AI model generates 3D models and textures.

[0543] The server temporarily stores the generated data.

[0544] Step 5:

[0545] The server sends the generated item data to the terminal and receives confirmation from the user. The user confirms the provided generation results on the terminal and proceeds to the next step.

[0546] Input: Generated item data

[0547] Output: Generated data provided to the user

[0548] Specific behavior:

[0549] The server sends the generated data to the terminal as an HTTP response.

[0550] The device displays the provided data.

[0551] Step 6:

[0552] The user checks the generated results and inputs correction instructions as needed. For example, they may request a correction such as "make it a lighter blue." The device then converts this new instruction back into JSON format and sends it to the server.

[0553] Input: User correction instructions

[0554] Output: Request to convert correction instructions to JSON format

[0555] Specific behavior:

[0556] The user enters correction instructions into the interface.

[0557] The device converts the correction instructions into JSON format and sends them to the server.

[0558] Step 7:

[0559] The server receives the modification instructions, inputs them into the generation AI model again, and regenerates the item. The item is modified based on the new instructions.

[0560] Input: Correction instruction data

[0561] Output: 3D model and texture data of the modified item

[0562] Specific behavior:

[0563] The server analyzes the correction instructions and re-inputs them into the generative AI model.

[0564] A generative AI model generates modified items.

[0565] The server saves the modified data.

[0566] Step 8:

[0567] If the user is satisfied with the final generated item, the server registers this item data in the metaverse space.

[0568] Input: Satisfied item data

[0569] Output: Registration data in the metaverse space

[0570] Specific behavior:

[0571] The user makes a final confirmation and indicates satisfaction.

[0572] The server registers the item data in the metaverse space.

[0573] Step 9:

[0574] Users can complete the decoration of a room by placing registered items in the metaverse space and adjusting their position and orientation.

[0575] Input: Registered item data

[0576] Output: Position and orientation of placed items

[0577] Specific behavior:

[0578] A user logs into the metaverse space.

[0579] The user arranges the items and adjusts their position and orientation.

[0580] Through this series of steps, users can easily create high-quality items without any specialized knowledge, repeatedly modify them until they meet their requirements, and finally place them in the Metaverse space.

[0581] (Application example 1)

[0582] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0583] Conventional methods for building metaverses and virtual stores require users to have specialized design knowledge, making them difficult for average users. Customizing and arranging items also requires a lot of time and effort, making efficient design difficult. Therefore, there was a demand for a system that would allow users to easily design high-quality virtual stores.

[0584] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0585] In this invention, the server includes means for processing requests received from users and generating input data for generating items, means for automatically generating items based on the input data using a generative AI model, means for providing the generated item data to the user and generating items again in response to confirmation and correction requests, means for registering and arranging the finalized item data in the metaverse space, and means for providing a visual interface for arranging and customizing the generated items in a virtual store. This enables users, even without specialized knowledge, to easily and efficiently design and customize high-quality virtual stores.

[0586] "Means for processing requests received from users and generating input data for item generation" is a function that analyzes requests entered by users and converts them into data in a format suitable for item generation.

[0587] "Means for automatically generating items based on the input data using a generative AI model" refers to a function that passes input data to an AI model and automatically generates items such as 3D models and textures based on that data.

[0588] "Means for providing the generated item data to the user and generating the item again in response to confirmation and correction requests" is a function that shows the generated item data to the user and, if necessary, generates the item again by reflecting correction requests made by the user.

[0589] The "means for registering and arranging the finalized item data in the metaverse space" is a function for registering the item data that the user has finally confirmed in the metaverse space and arranging it.

[0590] "Means for providing a visual interface for arranging and customizing items generated in a virtual store" is a function that provides a user with an interface that allows them to visually arrange and customize items generated in a virtual store.

[0591] The "format conversion means" is a means for converting a user request into input data.

[0592] The "process of generating a 3D model and texture" refers to the procedure by which an AI model generates a three-dimensional shape and its surface material based on a user's request.

[0593] A system for implementing the present invention automatically generates items for a virtual store based on user requests. This system includes a series of processes: a user inputs the request using a terminal, a server processes the request data and uses a generation AI model, a server provides the generated results and reflects correction instructions, and finally, the system registers and places the final item data in the virtual store.

[0594] First, a user uses a terminal to input a request for creating an item. For example, if a user wants to create a "wooden shelf" or a "retro lamp," the user inputs these requests through the interface. The terminal converts these requests into an appropriate format (e.g., JSON format) and sends them to the server.

[0595] Next, the server receives the request data sent by the user. The server analyzes this data and processes it as input data for the generative AI model. The generative AI model used by the server is generally composed of AI technologies such as GPT-4 and Stable Diffusion. The generative AI model generates a 3D model and texture of the item based on the input data. The generated item data is temporarily stored on the server.

[0596] The generated item data is sent from the server to the device for confirmation by the user. The user can check the generated results on the device and input correction instructions as needed. For example, a request for a "sofa with a lighter blue color" is possible. The server then uses the generative AI model again based on the new instructions to generate the corrected item. This allows the user to repeatedly make corrections until they achieve the desired result.

[0597] Finally, if the user is satisfied with the generated items, the server registers the item data in the virtual store space. The registered items become available for the user to place and use in the virtual store. The user adjusts the position and orientation of the items in their virtual store to complete the design of the store interior.

[0598] To give a specific example, when a user wants to generate a "wooden shelf," the following steps are taken: First, the user enters "wooden shelf" into the device interface. The device then converts this into JSON format and sends it to the server. The server receives the request data, passes "wooden shelf" as input to the generative AI model, and generates a 3D model and texture. The generated shelf data is sent to the device for confirmation by the user. If necessary, a correction request can be made, such as "make it a lighter color," and the corrected data is then generated again.

[0599] The above system configuration enables users to easily and efficiently design and customize high-quality virtual stores, even without specialized knowledge.

[0600] Example prompt sentences to use:

[0601] "Create a modern, high-quality blue sofa."

[0602] "Please change it to a lighter blue."

[0603] "Generate a wooden shelf."

[0604] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0605] Step 1:

[0606] The user inputs a request to create an item through the terminal interface. The user inputs a detailed prompt such as "wooden shelf" or "retro lamp." The input prompt is converted into an appropriate format (e.g., JSON format) by the terminal. Here, the user's request is the input, and the converted formatted data is the output.

[0607] Step 2:

[0608] The terminal transmits the format-converted data to the server. Specifically, the terminal uploads the converted data to the server via the network. Here, the format-converted data is the input, and the requested data received by the server is the output.

[0609] Step 3:

[0610] The server analyzes the received request data and creates input data for the generative AI model. Specifically, the server analyzes the JSON format data and formats it in a format that the generative AI model can understand. Here, the received request data is received as input, and the input data for the generative AI model is obtained as output.

[0611] Step 4:

[0612] The server uses a generative AI model to generate a 3D model and texture for the item. Specifically, the server passes input data to a generative AI model (such as GPT-4 or Stable Diffusion) and generates the item based on that data. Here, the input to the generative AI model is the input, and the generated 3D model and texture data are obtained as the output.

[0613] Step 5:

[0614] The server sends the generated 3D model and texture data to the terminal and provides it to the user. Specifically, the server sends the generated results to the terminal via the network. Here, the generated 3D model and texture data are the input, and the generated results displayed on the terminal are obtained as the output.

[0615] Step 6:

[0616] The user uses the terminal to check the generated results and input correction requests as necessary. The user may input a correction request such as "make the color a little brighter" into the terminal. Here, the generated item data and the user's correction request are input, and the correction request is sent from the terminal to the server as output.

[0617] Step 7:

[0618] The server receives the modification request and generates the modified item using the generative AI model again. Specifically, the server analyzes the modification request and passes it to the generative AI model again to generate a new 3D model and texture. Here, the modification request is the input and the modified 3D model and texture data are obtained as the output.

[0619] Step 8:

[0620] Finally, if the user is satisfied with the generated item, the server registers this item data in the virtual store space. Specifically, the server saves the item data in storage so that it can be placed in the user's virtual store. Here, the input is the finalized item data, and the output is the item registered in the virtual store.

[0621] Step 9:

[0622] Users use their devices to arrange and customize items in a virtual store. Specifically, they adjust the position and orientation of items using a visual interface. Here, the input is the items displayed in the virtual store, and the output is the final store design.

[0623] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0624] The system of the present invention not only automatically generates items in the metaverse space based on user requests, but also recognizes the user's emotions and suggests or modifies items according to those emotions. This system involves a series of processes: the user inputs the request using a terminal, the server processes the request data and uses a generation AI model and emotion engine, the server provides the generated results and reflects modification instructions, and the final item data is registered and placed in the metaverse space.

[0625] First, a user uses a terminal to input a request for creating an item. For example, if a user wants to create a "blue modern sofa," the user inputs detailed instructions such as "sofa," "blue," and "modern" into the interface. The terminal converts these requests into an appropriate format (e.g., JSON format) and sends them to the server.

[0626] Next, the server receives the request data sent by the user. The server analyzes this data and processes it as input data for the generative AI model and emotion engine. The generative AI model generates a 3D model and texture for the item based on this input data. The emotion engine also recognizes the user's emotion and adjusts the parameters of the generative AI model to generate an item that best suits the user's emotion. The generated item data is temporarily stored on the server.

[0627] The generated item data is sent from the server to the device for user confirmation. The user can check the generated results on the device and input correction instructions as necessary. For example, if the color of a sofa differs from what the user's emotions were considered, the emotion engine analyzes the user's emotions from their facial expressions, voice, and text input and suggests appropriate corrections. If the user accepts the suggestions, the device reformats the correction instructions in JSON format and sends them to the server.

[0628] The server uses the generative AI model based on new instructions and the user's emotional data to generate a revised item. This allows the user to repeatedly make revisions until they achieve the desired result. Finally, if the user is satisfied with the generated item, the server registers this item data in the metaverse space. The registered item can then be placed and used by the user within the metaverse space. The user can then adjust the position and orientation of the item within their metaverse space to complete the decoration of their room.

[0629] Specific examples

[0630] The specific flow when a user creates a "blue modern sofa" is shown below.

[0631] 1. Request Input

[0632] The user types "sofa," "blue," and "modern" into the device interface.

[0633] The terminal converts this into JSON format and sends it to the server.

[0634] 2. Processing of request data and use of generation AI models and emotion engines

[0635] The server receives and analyzes the request data.

[0636] The generative AI model is given the inputs "sofa," "blue," and "modern," and generates a 3D model and texture of a blue modern sofa.

[0637] The emotion engine analyzes the user's facial expressions and voice and adjusts the parameters of the generative AI model.

[0638] 3. Providing and Modifying Generated Results

[0639] The server transmits the generated sofa data to the terminal.

[0640] The user checks the result and inputs a correction request such as "make it a little brighter blue."

[0641] The emotion engine analyzes the user's emotions and suggests the most suitable colors.

[0642] The server inputs the modification request back into the generative AI model and generates the modified sofa.

[0643] 4. Registration and placement in the metaverse space

[0644] Finally, the server registers the sofa that the user is satisfied with in the metaverse space.

[0645] The user places the sofa in the metaverse space and completes the decoration.

[0646] This system allows users to easily generate and arrange high-quality content without specialized knowledge, and also provides optimal items that take into account the user's emotions.

[0647] The processing flow will be explained below.

[0648] Step 1:

[0649] The user enters a request

[0650] The user uses the device interface to input a request for the item they want to create. For example, to specify a "blue modern sofa," they input "sofa," "blue," and "modern." The device then converts this request into JSON-formatted data.

[0651] Step 2:

[0652] The device sends the requested data to the server

[0653] The terminal sends the converted request data to the server as an HTTP request. The server analyzes the received request data and converts it into a format that can be handled internally.

[0654] Step 3:

[0655] The server inputs data into the generated AI model and emotion engine

[0656] The server passes the analyzed request data to the generative AI model and emotion engine as input data. The generative AI model generates a 3D model and texture of the requested item based on the input data. Meanwhile, the emotion engine collects and analyzes data such as the user's facial expressions, voice, and text input to analyze the user's emotions.

[0657] Step 4:

[0658] Generative AI model and emotion engine generate items and emotion data

[0659] The generative AI model automatically generates a 3D model and texture of a blue modern sofa based on the specifications of "sofa," "blue," and "modern." Meanwhile, the emotion engine analyzes the user's input data and real-time emotion data to suggest the optimal items to satisfy the user. The generated item data and emotion data are returned to the server.

[0660] Step 5:

[0661] The server returns the generated results to the terminal

[0662] The server organizes the generated item data and the analysis results of the emotion engine and sends them back to the device, which then displays the generated results and suggestions to the user.

[0663] Step 6:

[0664] The user can review the generated results and suggestions and enter correction requests if necessary.

[0665] The user checks the generated items and the emotion engine's suggestions on the device. For example, if the color is different from what they expected, the user can input a correction request into the device, such as "make it a lighter blue." The device then converts this into JSON format and sends it to the server.

[0666] Step 7:

[0667] The server generates a correction request again and inputs it into the AI ​​model and emotion engine.

[0668] The server receives the user's modification request and again inputs the data into the generative AI model and emotion engine. The generative AI model generates a modified 3D model and texture based on the new input data, and the emotion engine generates new emotion data and suggestions. The generated data is returned to the server and sent back to the device.

[0669] Step 8:

[0670] The user confirms the item and confirms it

[0671] The user reviews the results again and repeats steps 6 and 7 as necessary until satisfied, finally finalizing the satisfactory items.

[0672] Step 9:

[0673] The server registers the item data in the metaverse space.

[0674] The server registers the confirmed item data in the metaverse space database. The item is reflected in the user's metaverse space.

[0675] Step 10:

[0676] Users place items in the metaverse space

[0677] The user logs in to the metaverse space using a terminal and places the generated items wherever they like. The user adjusts the position and orientation of the items to complete the final layout.

[0678] Example 2

[0679] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0680] In conventional systems, it was difficult for users to generate specific items while taking their emotions into consideration and providing optimal items. Furthermore, reflecting requests for revisions to generated items required manual work and multiple trial and error processes, which was inefficient. There was also a need for a system that could centrally manage these processes and be intuitive for users to operate.

[0681] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0682] In this invention, the server includes means for processing requests received from users and generating input data for generation, means for automatically generating items based on the input data using a generative model, means for providing the generated item data to the user and generating the item again in response to a request for confirmation and correction, means for recognizing the user's emotions and adjusting the parameters of the generative model in accordance with the emotions, and means for registering and arranging the finalized item data in a virtual space. This allows users to efficiently generate and modify optimal items that reflect their emotions and arrange them in a virtual space.

[0683] "User" refers to a person who utilizes a terminal to input requests into the system and to review and modify generated items.

[0684] "Server" refers to a computer system that receives request data from a user and generates and modifies items using generative models and emotion recognition engines.

[0685] A "terminal" refers to a device operated by a user to input requests and communicate with a server. Specifically, this applies to smartphones and computers.

[0686] "Request data" refers to data obtained by converting detailed information about an item that a user inputs through a terminal into a different format.

[0687] A "generative model" refers to an artificial intelligence model that automatically generates 3D models and textures based on required data.

[0688] An "emotion recognition engine" refers to a software system that analyzes a user's facial expressions and voice to identify the user's emotions.

[0689] "Virtual space" refers to a three-dimensional computer-generated environment constructed using digital technology that users can manipulate and experience as if it were a physical space through an interface.

[0690] "Input data" refers to data used by the generative model based on user request data.

[0691] The system of the present invention automatically generates items in a virtual space based on user requests, recognizes the user's emotions, and suggests or modifies items according to the emotions. This system includes a series of processes: request input, request data processing, use of a generation AI model and an emotion recognition engine, provision of the generated results and reflection of modification requests, and final registration and placement of the item data in the virtual space.

[0692] First, a user uses a terminal to input a request for creating an item. For example, if a user wants to create a "blue modern sofa," the user inputs detailed instructions such as "sofa," "blue," and "modern" into the interface. The terminal converts these requests into an appropriate format (e.g., JSON format) and sends them to the server.

[0693] The server receives and analyzes the request data sent by the user. The analyzed data is passed to the generative AI model and emotion recognition engine. The generative AI model generates a 3D model and texture of the item based on this input data. Meanwhile, the emotion recognition engine recognizes the user's emotion and adjusts the parameters of the generative AI model. For example, if the user is smiling, the emotion recognition engine instructs the generated item to use bright colors. The generated item data is temporarily stored on the server.

[0694] The generated item data is sent from the server to the device for user confirmation. The user can review the generated results on the device and input correction requests as needed. For example, if the color of the sofa is darker than expected, the user can input a correction request such as "make it a lighter blue." The emotion recognition engine analyzes the user's emotions and the new request and suggests the optimal correction. The server then uses the generative AI model again based on the new instructions to generate the corrected item.

[0695] Finally, if the user is satisfied with the generated item, the server registers the item data in the virtual space. The registered item can then be placed and used by the user in the virtual space. The user can then adjust the position and orientation of the item in their virtual space to complete the room decoration.

[0696] Specific examples

[0697] A specific example prompt for a user to generate a "blue modern sofa" would be:

[0698] "Generate a blue modern sofa."

[0699] "Please suggest a blue sofa that reflects the emotion you read from the user's facial expression."

[0700] This system allows users to easily create and arrange high-quality items without specialized knowledge, and also provides optimal items that take into account the user's emotions.

[0701] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0702] Step 1: User enters item creation request

[0703] The user inputs details of the item they want to create (e.g., "sofa," "blue," "modern") into the device interface. The device receives this as input data and converts it into JSON format using a format conversion tool. The input is the user's instructions, and the output is JSON format data.

[0704] Step 2: The terminal converts the requested data into a different format and sends it

[0705] The terminal converts the item creation request entered by the user into JSON format and then sends this data to the server. Here, the input is the detailed request data from the user, and the output is the JSON data sent to the server.

[0706] Step 3: Parsing and processing the request data by the server

[0707] The server receives the JSON data sent from the device and analyzes it. The analyzed content includes the item type, color, style, etc. The analyzed data is passed as input to the generative AI model and emotion recognition engine. The input is JSON data, and the output is the analyzed item information.

[0708] Step 4: Generate items using a generative AI model

[0709] The server passes the analyzed item information to the generative AI model, which generates a 3D model and texture according to the specified conditions (e.g., "blue modern sofa"). The generated data is temporarily stored on the server. The input is the analyzed item information, and the output is the generated 3D model and texture.

[0710] Step 5: Recognize and adjust the user's emotions using the emotion recognition engine

[0711] The server uses an emotion recognition engine to recognize the user's emotions. This involves analyzing the user's facial expressions and voice using the device's camera and microphone. The input is the user's facial expression and voice data, and the output is the recognized emotional information. This emotional information is used to adjust the parameters of the generative AI model and reflect it in the output.

[0712] Step 6: Server saves and sends the generated results

[0713] The server temporarily stores the generated and adjusted item data and then transmits it to the user's device. The input is the adjusted item data, and the output is the data sent to the device.

[0714] Step 7: User checks the generated results and requests corrections

[0715] The user checks the generated item on the terminal interface and inputs correction requests as needed. For example, the user may request "make it a lighter blue." The input is the generated item data, and the output is the user's correction requests.

[0716] Step 8: Emotion Recognition Engine Suggests Corrections

[0717] The emotion recognition engine proposes optimal corrections based on the user's new request and the results of facial and voice analysis. The input is the user's new request and emotional data, and the output is the optimal correction proposal.

[0718] Step 9: Server creates and resubmits the corrected item

[0719] The server uses the generative AI model again based on the user's correction request to generate a corrected item. This corrected item is then resent to the user's device. The input is the correction instruction and emotion data, and the output is the corrected item data.

[0720] Step 10: Register and place items in the metaverse space

[0721] The server registers the items that the user is finally satisfied with in the virtual space. After registration is complete, the user can freely place items in the virtual space and customize the environment. The input is the finalized item data, and the output is the items that have been registered in the virtual space and can now be placed.

[0722] (Application example 2)

[0723] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0724] Previously, when creating items in a virtual store or metaverse space, users had to manually input and edit data using a limited interface, which was extremely time-consuming and laborious. Furthermore, if the created item did not match the user's emotions or preferences, the user would have to re-edit it, resulting in a poor user experience. The present invention aims to solve these problems and provide a system that allows users to create and edit items more comfortably and efficiently.

[0725] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing the user's emotions in real time and adjusting the parameters of the generation AI model based on the emotion data, means for providing the generated item data to the user and generating items again in response to confirmation and correction requests, and means for registering and arranging the finalized item data in the virtual space. This allows the user to easily generate and modify optimal items based on their emotions and preferences, and efficiently arrange them.

[0726] "Means for recognizing a user's emotions in real time and adjusting the parameters of a generative AI model based on the emotion data" refers to a system that has the ability to recognize emotions in real time not only from data entered by the user but also from the user's facial expressions, voice, etc., and dynamically adjust the operating parameters of a generative AI model according to those emotions.

[0727] A "generative AI model" is an artificial intelligence model that automatically generates specific items based on input data, including processes that generate 3D models and textures.

[0728] The "emotion engine" analyzes the user's emotions and suggests or modifies items based on that emotion data, thereby providing the optimal items that the user desires.

[0729] "Virtual space" refers to a virtual area where users can engage in various activities within a digital space, such as a metaverse environment or a virtual store provided via the Internet.

[0730] The "means for generating an item again in response to a confirmation and correction request" is a system function that, when a user confirms a generated item and requests correction as necessary, generates the item again based on that request.

[0731] "Means for registering and arranging finalized item data in a virtual space" is a function that allows a user to register items that have been confirmed to their satisfaction in a virtual space and arrange them freely within that virtual space.

[0732] The "format conversion means for converting a user's request into input data" is a means having the function of appropriately formatting a request input by a user and converting it into a form that is easily understood by the system.

[0733] The system according to the present invention automatically generates items in a virtual space based on requests from a user, and further recognizes the user's emotions and suggests or modifies items according to the emotions.

[0734] System Program

[0735] Basic configuration

[0736] User terminal: A device used by users to input requests and check generated items. This can be a smart glass or a smartphone.

[0737] Server: The central device that processes request data and generates items using generative AI models and emotion engines.

[0738] Generative AI model: An artificial intelligence model that generates items based on user requests.

[0739] Emotion engine: An engine that analyzes user emotion data and adjusts the parameters of the generative AI model based on that data.

[0740] Explanation of program processing

[0741] The server first receives a request entered by the user via their device. This request is converted into a data format such as JSON. The server then inputs the converted data into a generative AI model to generate a 3D model and texture of the item based on the request. The emotion engine also recognizes emotional data from the user's facial expressions, voice, and text input, and adjusts the parameters of the generative AI model.

[0742] The generated item is immediately sent to the user's terminal, where the user can confirm it. If the user requires any modifications, the request is sent again to the server. This allows the optimal item to be generated according to the user's feelings. Modifications continue until the user is satisfied, and the finalized item is registered in the virtual space.

[0743] Specific examples

[0744] Suppose a user accesses a virtual store using smart glasses and enters a request by voice, such as "a blue modern sofa." The keywords "sofa," "blue," and "modern" are transmitted to the server through the interface of the smart glasses.

[0745] The server receives this and generates a 3D model and texture of a "blue modern sofa." At the same time, the emotion engine detects a "relaxed atmosphere" from the user's smile and adjusts the color of the sofa slightly brighter accordingly. The generated result is presented to the user, and if the user requests a correction such as "a darker blue," the item is generated again using the same process.

[0746] Example prompt sentence:

[0747] "Create a modern blue sofa that has a relaxing atmosphere."

[0748] This system allows users to easily create items that best suit their emotions and preferences, and efficiently place them in the virtual space.

[0749] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0750] Step 1:

[0751] A user uses a device (such as smart glasses or a smartphone) to input requests such as "sofa," "blue," and "modern" into the user interface. This input request data is converted into a data format such as JSON and sent to the server.

[0752] Input: User request (e.g. "sofa", "blue", "modern")

[0753] Output: Formatted request data (e.g., JSON format)

[0754] Specific operation:

[0755] When a user inputs something using voice or gestures, the device analyzes it and converts it into text data, which it then converts into JSON format and sends to the server via the network.

[0756] Step 2:

[0757] The server receives and analyzes the request data sent from the device. The analyzed data is input into the generative AI model to generate a 3D model and texture of a "blue modern sofa." The generated temporal data is also adjusted by the emotion engine based on the user's emotional data.

[0758] Input: Formatted request data

[0759] Output: Generated item data (3D models and textures)

[0760] Specific operation:

[0761] The server analyzes the received JSON data and passes the keywords "sofa," "blue," and "modern" to the generative AI model. The generative AI model creates a 3D model and texture based on the data. At the same time, the emotion engine analyzes the user's emotion data and adjusts the parameters of the generative AI model to generate the optimal output.

[0762] Step 3:

[0763] The generated item data is temporarily stored and sent to the user's terminal. The user can check the generated item on the terminal and, if not satisfied, input a correction request. This correction request is sent back to the server.

[0764] Input: Generated item data

[0765] Output: Correction requests from the user

[0766] Specific operation:

[0767] The item data is sent from the server to the device, where the user confirms it. If the user wishes to make any changes, they can enter their request by voice or text, such as "make it a lighter blue." The device then converts this request back into JSON format and sends it to the server.

[0768] Step 4:

[0769] The server receives the modification request from the user again and generates the item again using the generative AI model and emotion engine, thereby providing an item optimized for the user's emotions.

[0770] Input: User request for correction (e.g., "Make it a lighter blue")

[0771] Output: Regenerated item data

[0772] Specific operation:

[0773] The server receives the correction request, inputs it back into the generative AI model as data, and uses the emotion engine to readjust it based on the user's emotions. The generative AI model then creates new 3D models and textures and sends the data to the user's device.

[0774] Step 5:

[0775] Finally, once an item that satisfies the user is created, the server registers and places the item data in the virtual space. The user can then adjust the position and orientation of the item within the virtual space to complete the decoration of the space.

[0776] Input: Finalized item data

[0777] Output: Items registered in the virtual space

[0778] Specific operation:

[0779] The server receives the item data that the user is ultimately satisfied with and registers it in the specified location within the virtual space. The user can then freely adjust the position and orientation of the items within the virtual space via their device to complete the decoration of the space.

[0780] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0781] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0782] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0783] [Third embodiment]

[0784] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0785] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0786] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0787] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0788] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0789] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0790] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0791] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0792] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0793] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0794] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0795] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0796] The system of the present invention automatically generates items in the Metaverse space based on user requests. This system includes a series of processes: the user inputs the request using a terminal, the server processes the request data and uses a generation AI model, the server provides the generated results and reflects correction instructions, and the final item data is registered and placed in the Metaverse space.

[0797] First, a user uses a terminal to input a request for creating an item. For example, if a user wants to create a "blue modern sofa," the user inputs detailed instructions such as "sofa," "blue," and "modern" into the interface. The terminal converts these requests into an appropriate format (e.g., JSON format) and sends them to the server.

[0798] Next, the server receives the request data sent by the user. The server analyzes this data and processes it as input data for the generative AI model. The generative AI model generates a 3D model and texture of the item based on this input data. The generated item data is temporarily stored on the server.

[0799] The generated item data is sent from the server to the device for confirmation by the user. The user can check the generated results provided on the device and input correction instructions as necessary. For example, if the user wants to change the color of the sofa to a "lighter blue," they can send that instruction again from the device to the server. The server will then use the generative AI model again based on the new instructions to generate the corrected item. This allows the user to make repeated corrections until they achieve the desired result.

[0800] Finally, if the user is satisfied with the created item, the server registers this item data in the metaverse space. The registered item can then be placed and used by the user in the metaverse space. The user can then adjust the position and orientation of the item in their metaverse space to complete the room decoration.

[0801] Specific examples

[0802] The specific flow when a user creates a "blue modern sofa" is shown below.

[0803] 1. Request Input

[0804] The user types "sofa," "blue," and "modern" into the device interface.

[0805] The terminal converts this into JSON format and sends it to the server.

[0806] 2. Processing requirements data and using generative AI models

[0807] The server receives and analyzes the request data.

[0808] The generative AI model is given the inputs "sofa," "blue," and "modern," and generates a 3D model and texture of a blue modern sofa.

[0809] 3. Providing and Modifying Generated Results

[0810] The server transmits the generated sofa data to the terminal.

[0811] The user checks the result and inputs a correction request such as "make it a little brighter blue."

[0812] The server inputs the modification request back into the generative AI model and generates the modified sofa.

[0813] 4. Registration and placement in the metaverse space

[0814] Finally, the server registers the sofa that the user is satisfied with in the metaverse space.

[0815] The user places the sofa in the metaverse space and completes the decoration.

[0816] The system allows users to easily generate and distribute high-quality content without specialized knowledge.

[0817] The processing flow will be explained below.

[0818] Step 1:

[0819] The user enters a request

[0820] The user uses the device interface to input a request for the item they want to create. For example, to specify a "blue modern sofa," they input "sofa," "blue," and "modern." The device then converts this request into JSON-formatted data.

[0821] Step 2:

[0822] The device sends the requested data to the server

[0823] The terminal sends the converted request data to the server as an HTTP request. The server analyzes the received request data and converts it into a format that can be handled internally.

[0824] Step 3:

[0825] The server inputs data into the generative AI model

[0826] The server passes the analyzed request data as input to the generative AI model, which then generates a 3D model and texture for the requested item based on the input data.

[0827] Step 4:

[0828] Generative AI model generates items

[0829] The generative AI model automatically generates a 3D model and texture of a blue modern sofa based on the specifications of "sofa," "blue," and "modern." The generated item data is then returned to the server.

[0830] Step 5:

[0831] The server returns the generated results to the terminal

[0832] The server organizes the item data returned by the generative AI model and sends it back to the device, which then displays the generated results to the user.

[0833] Step 6:

[0834] The user checks the generated results and inputs correction requests as necessary.

[0835] The user checks the generated item on the device. For example, if the color is different from what they expected, they input a correction request into the device, such as "make it a lighter blue." The device then reformats this into JSON and sends it to the server.

[0836] Step 7:

[0837] The server inputs the correction request into the generated AI model again.

[0838] The server receives the user's modification request and inputs the data back into the generative AI model, which then generates a new item based on the modification request.

[0839] Step 8:

[0840] The generative AI model generates the modified item.

[0841] The generative AI model generates a modified 3D model and texture based on the new input data and returns it to the server, which then sends the modified item data back to the device.

[0842] Step 9:

[0843] The user confirms the item and confirms it

[0844] The user reviews the results again and repeats steps 6 through 8 as necessary until satisfied, finally finalizing the items they are satisfied with.

[0845] Step 10:

[0846] The server registers the item data in the metaverse space.

[0847] The server registers the confirmed item data in the metaverse space database. The item is reflected in the user's metaverse space.

[0848] Step 11:

[0849] Users place items in the metaverse space

[0850] The user logs in to the metaverse space using a terminal and places the generated items wherever they like. The user adjusts the position and orientation of the items to complete the final layout.

[0851] Example 1

[0852] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0853] In the Metaverse, it has been difficult for users to easily generate high-quality items and place them as desired. Conventional systems require advanced expertise and complex operations, placing a heavy burden on many users. In addition, there is also the problem that if a generated item does not completely meet the user's requirements, it is not easy to regenerate or modify it.

[0854] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0855] In this invention, the server includes means for processing requests received from users and generating input data for generating items, means for converting the user's request into JSON format using a terminal and transmitting the JSON data to the server, means for analyzing the JSON data received by the server and formatting it as input data for the generative AI model, means for automatically generating items based on the input data using the generative AI model, means for temporarily storing the generated item data on the server, means for providing the generated item data to the user and generating items again in response to the user's confirmation and correction requests, means for registering and arranging item data that the user is satisfied with in the metaverse space, and means for adjusting the position and orientation of registered items in the metaverse space. This enables users without specialized knowledge to easily generate high-quality items, make corrections until they meet their requirements, and finally place them in the metaverse space.

[0856] "User" refers to an individual or organization that uses this system to generate and place items.

[0857] "Terminal" refers to a computer or mobile device used by a User to register input data and view generated items.

[0858] "Server" refers to a computer system that receives user requests, analyzes the request data, generates items using a generative AI model, and registers the item data in the Metaverse space.

[0859] "Means for generating input data" refers to the function of converting a request received from a user into the data format (e.g., JSON format) required for item generation.

[0860] "JSON format" is an abbreviation for JavaScript Object Notation, and refers to a lightweight data exchange format for expressing data in a format that is easy for humans to read and machines to parse.

[0861] "Generative AI model" refers to an artificial intelligence model that automatically generates items such as 3D models and textures based on input data.

[0862] "Item Data" refers to digital data, including 3D models and textures, generated by a generative AI model.

[0863] The "metaverse space" is a virtual digital world, an online environment where users can place and interact with registered items.

[0864] "Means for adjusting position and orientation" refers to the function that allows users to change the position and orientation of registered items in the metaverse space and optimize their placement.

[0865] "Means for generating an item again in response to confirmation and correction requests" refers to the function that, when the user confirms the generated results provided and inputs a correction request, regenerates the item using the generation AI model again based on that request.

[0866] The system of the present invention automatically generates items in the Metaverse space based on user requests. This system includes a series of processes: the user inputs the request using a terminal, the server processes the request data and uses a generation AI model, the server provides the generated results and reflects correction instructions, and the final item data is registered and placed in the Metaverse space.

[0867] First, a user uses a terminal to input a request for creating an item. For example, if a user wants to create a "blue modern sofa," the user inputs detailed instructions such as "sofa," "blue," and "modern" into the interface. The terminal converts these requests into an appropriate format (e.g., JSON format) and sends them to the server.

[0868] Next, the server receives the request data sent by the user. The server analyzes this data and formats it as input data for the generative AI model. The generative AI model generates a 3D model and texture of the item based on this input data. The generated item data is temporarily stored on the server.

[0869] The generated item data is sent from the server to the device for confirmation by the user. The user can check the generated results provided on the device and input correction instructions as necessary. For example, if the user wants to change the color of the sofa to a "lighter blue," they can send that instruction again from the device to the server. The server will then use the generative AI model again based on the new instructions to generate the corrected item. This allows the user to make repeated corrections until they achieve the desired result.

[0870] Finally, if the user is satisfied with the created item, the server registers this item data in the metaverse space. The registered item can then be placed and used by the user in the metaverse space. The user can then adjust the position and orientation of the item in their metaverse space to complete the room decoration.

[0871] Hardware and software used

[0872] Server: A high-performance server (e.g., a server equipped with an NVIDIA GPU)

[0873] Generative AI model: Examples of models include OpenAI's DALL-E and Google's DeepMind

[0874] Data format: JSON

[0875] Specific examples

[0876] The specific flow when a user creates a "blue modern sofa" is shown below.

[0877] 1. Request Input

[0878] The user types "sofa," "blue," and "modern" into the device interface.

[0879] The terminal converts this into JSON format and sends it to the server.

[0880] 2. Processing requirements data and using generative AI models

[0881] The server receives and analyzes the request data.

[0882] The generative AI model is given the inputs "sofa," "blue," and "modern," and generates a 3D model and texture of a blue modern sofa.

[0883] 3. Providing and Modifying Generated Results

[0884] The server transmits the generated sofa data to the terminal.

[0885] The user checks the result and inputs a correction request such as "make it a little brighter blue."

[0886] The server inputs the modification request back into the generative AI model and generates the modified sofa.

[0887] 4. Registration and placement in the metaverse space

[0888] Finally, the server registers the sofa that the user is satisfied with in the metaverse space.

[0889] The user places the sofa in the metaverse space and completes the decoration.

[0890] Prompt Sentence Examples

[0891] Here are some example input prompts for a generative AI model:

[0892] Example prompt

[0893] "Sofa" "Blue" "Modern"

[0894] The system allows users to easily generate and distribute high-quality content without requiring specialized knowledge.

[0895] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0896] Step 1:

[0897] The user inputs a request for item creation into the terminal interface. The user inputs a specific request, such as "sofa," "blue," and "modern," into the terminal. This causes the terminal to prepare the data for step 2.

[0898] Input: User request (e.g. "sofa", "blue", "modern")

[0899] Output: Request to convert input data to JSON format

[0900] Specific behavior:

[0901] The user enters a keyword into the input field on the device.

[0902] The terminal receives input data.

[0903] Step 2:

[0904] The terminal converts the user's request into JSON format. The converted JSON data is sent to the server. At this stage, the data format is unified and formatted so that it can be passed to the server.

[0905] Input: User request data

[0906] Output: Data converted to JSON format

[0907] Specific behavior:

[0908] The terminal converts the input data into JSON format.

[0909] The terminal sends the converted JSON data to the server as an HTTP request.

[0910] Step 3:

[0911] The server parses the received JSON data and formats it as input data for the generative AI model. The JSON data is parsed and converted into a prompt that the AI ​​model can understand.

[0912] Input: User request data in JSON format

[0913] Output: Input data to the generative AI model

[0914] Specific behavior:

[0915] The server deserializes and parses the JSON data.

[0916] The server formats the data as required.

[0917] Step 4:

[0918] The server inputs the formatted data into a generative AI model to generate a 3D model and texture for the item. The generative AI model generates the item based on the input data, and the data is temporarily stored on the server.

[0919] Input: Formatted input data

[0920] Output: 3D model and texture data of the generated item

[0921] Specific behavior:

[0922] The server sends the formatted data to the generative AI model.

[0923] Generative AI model generates 3D models and textures.

[0924] The server temporarily stores the generated data.

[0925] Step 5:

[0926] The server sends the generated item data to the terminal and receives confirmation from the user. The user confirms the provided generation results on the terminal and proceeds to the next step.

[0927] Input: Generated item data

[0928] Output: Generated data provided to the user

[0929] Specific behavior:

[0930] The server sends the generated data to the terminal as an HTTP response.

[0931] The device displays the provided data.

[0932] Step 6:

[0933] The user checks the generated results and inputs correction instructions as needed. For example, they may request a correction such as "make it a lighter blue." The device then converts this new instruction back into JSON format and sends it to the server.

[0934] Input: User correction instructions

[0935] Output: Request to convert correction instructions to JSON format

[0936] Specific behavior:

[0937] The user enters correction instructions into the interface.

[0938] The device converts the correction instructions into JSON format and sends them to the server.

[0939] Step 7:

[0940] The server receives the modification instructions, inputs them into the generation AI model again, and regenerates the item. The item is modified based on the new instructions.

[0941] Input: Correction instruction data

[0942] Output: 3D model and texture data of the modified item

[0943] Specific behavior:

[0944] The server analyzes the correction instructions and re-inputs them into the generative AI model.

[0945] A generative AI model generates modified items.

[0946] The server saves the modified data.

[0947] Step 8:

[0948] If the user is satisfied with the final generated item, the server registers this item data in the metaverse space.

[0949] Input: Satisfied item data

[0950] Output: Registration data in the metaverse space

[0951] Specific behavior:

[0952] The user makes a final confirmation and indicates satisfaction.

[0953] The server registers the item data in the metaverse space.

[0954] Step 9:

[0955] Users can complete the decoration of a room by placing registered items in the metaverse space and adjusting their position and orientation.

[0956] Input: Registered item data

[0957] Output: Position and orientation of placed items

[0958] Specific behavior:

[0959] A user logs into the metaverse space.

[0960] The user arranges the items and adjusts their position and orientation.

[0961] Through this series of steps, users can easily create high-quality items without any specialized knowledge, repeatedly modify them until they meet their requirements, and finally place them in the Metaverse space.

[0962] (Application example 1)

[0963] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0964] Conventional methods for building metaverses and virtual stores require users to have specialized design knowledge, making them difficult for average users. Customizing and arranging items also requires a lot of time and effort, making efficient design difficult. Therefore, there was a demand for a system that would allow users to easily design high-quality virtual stores.

[0965] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0966] In this invention, the server includes means for processing requests received from users and generating input data for generating items, means for automatically generating items based on the input data using a generative AI model, means for providing the generated item data to the user and generating items again in response to confirmation and correction requests, means for registering and arranging the finalized item data in the metaverse space, and means for providing a visual interface for arranging and customizing the generated items in a virtual store. This enables users, even without specialized knowledge, to easily and efficiently design and customize high-quality virtual stores.

[0967] "Means for processing requests received from users and generating input data for item generation" is a function that analyzes requests entered by users and converts them into data in a format suitable for item generation.

[0968] "Means for automatically generating items based on the input data using a generative AI model" refers to a function that passes input data to an AI model and automatically generates items such as 3D models and textures based on that data.

[0969] "Means for providing the generated item data to the user and generating the item again in response to confirmation and correction requests" is a function that shows the generated item data to the user and, if necessary, generates the item again by reflecting correction requests made by the user.

[0970] The "means for registering and arranging the finalized item data in the metaverse space" is a function for registering the item data that the user has finally confirmed in the metaverse space and arranging it.

[0971] "Means for providing a visual interface for arranging and customizing items generated in a virtual store" is a function that provides a user with an interface that allows them to visually arrange and customize items generated in a virtual store.

[0972] The "format conversion means" is a means for converting a user request into input data.

[0973] The "process of generating a 3D model and texture" refers to the procedure by which an AI model generates a three-dimensional shape and its surface material based on a user's request.

[0974] A system for implementing the present invention automatically generates items for a virtual store based on user requests. This system includes a series of processes: a user inputs the request using a terminal, a server processes the request data and uses a generation AI model, a server provides the generated results and reflects correction instructions, and finally, the system registers and places the final item data in the virtual store.

[0975] First, a user uses a terminal to input a request for creating an item. For example, if a user wants to create a "wooden shelf" or a "retro lamp," the user inputs these requests through the interface. The terminal converts these requests into an appropriate format (e.g., JSON format) and sends them to the server.

[0976] Next, the server receives the request data sent by the user. The server analyzes this data and processes it as input data for the generative AI model. The generative AI model used by the server is generally composed of AI technologies such as GPT-4 and Stable Diffusion. The generative AI model generates a 3D model and texture of the item based on the input data. The generated item data is temporarily stored on the server.

[0977] The generated item data is sent from the server to the device for confirmation by the user. The user can check the generated results on the device and input correction instructions as needed. For example, a request for a "sofa with a lighter blue color" is possible. The server then uses the generative AI model again based on the new instructions to generate the corrected item. This allows the user to repeatedly make corrections until they achieve the desired result.

[0978] Finally, if the user is satisfied with the generated items, the server registers the item data in the virtual store space. The registered items become available for the user to place and use in the virtual store. The user adjusts the position and orientation of the items in their virtual store to complete the design of the store interior.

[0979] To give a specific example, when a user wants to generate a "wooden shelf," the following steps are taken: First, the user enters "wooden shelf" into the device interface. The device then converts this into JSON format and sends it to the server. The server receives the request data, passes "wooden shelf" as input to the generative AI model, and generates a 3D model and texture. The generated shelf data is sent to the device for confirmation by the user. If necessary, a correction request can be made, such as "make it a lighter color," and the corrected data is then generated again.

[0980] The above system configuration enables users to easily and efficiently design and customize high-quality virtual stores, even without specialized knowledge.

[0981] Example prompt sentences to use:

[0982] "Create a modern, high-quality blue sofa."

[0983] "Please change it to a lighter blue."

[0984] "Generate a wooden shelf."

[0985] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0986] Step 1:

[0987] The user inputs a request to create an item through the terminal interface. The user inputs a detailed prompt such as "wooden shelf" or "retro lamp." The input prompt is converted into an appropriate format (e.g., JSON format) by the terminal. Here, the user's request is the input, and the converted formatted data is the output.

[0988] Step 2:

[0989] The terminal transmits the format-converted data to the server. Specifically, the terminal uploads the converted data to the server via the network. Here, the format-converted data is the input, and the requested data received by the server is the output.

[0990] Step 3:

[0991] The server analyzes the received request data and creates input data for the generative AI model. Specifically, the server analyzes the JSON format data and formats it in a format that the generative AI model can understand. Here, the received request data is received as input, and the input data for the generative AI model is obtained as output.

[0992] Step 4:

[0993] The server uses a generative AI model to generate a 3D model and texture for the item. Specifically, the server passes input data to a generative AI model (such as GPT-4 or Stable Diffusion) and generates the item based on that data. Here, the input to the generative AI model is the input, and the generated 3D model and texture data are obtained as the output.

[0994] Step 5:

[0995] The server sends the generated 3D model and texture data to the terminal and provides it to the user. Specifically, the server sends the generated results to the terminal via the network. Here, the generated 3D model and texture data are the input, and the generated results displayed on the terminal are obtained as the output.

[0996] Step 6:

[0997] The user uses the terminal to check the generated results and input correction requests as necessary. The user may input a correction request such as "make the color a little brighter" into the terminal. Here, the generated item data and the user's correction request are input, and the correction request is sent from the terminal to the server as output.

[0998] Step 7:

[0999] The server receives the modification request and generates the modified item using the generative AI model again. Specifically, the server analyzes the modification request and passes it to the generative AI model again to generate a new 3D model and texture. Here, the modification request is the input and the modified 3D model and texture data are obtained as the output.

[1000] Step 8:

[1001] Finally, if the user is satisfied with the generated item, the server registers this item data in the virtual store space. Specifically, the server saves the item data in storage so that it can be placed in the user's virtual store. Here, the input is the finalized item data, and the output is the item registered in the virtual store.

[1002] Step 9:

[1003] Users use their devices to arrange and customize items in a virtual store. Specifically, they adjust the position and orientation of items using a visual interface. Here, the input is the items displayed in the virtual store, and the output is the final store design.

[1004] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1005] The system of the present invention not only automatically generates items in the metaverse space based on user requests, but also recognizes the user's emotions and suggests or modifies items according to those emotions. This system involves a series of processes: the user inputs the request using a terminal, the server processes the request data and uses a generation AI model and emotion engine, the server provides the generated results and reflects modification instructions, and the final item data is registered and placed in the metaverse space.

[1006] First, a user uses a terminal to input a request for creating an item. For example, if a user wants to create a "blue modern sofa," the user inputs detailed instructions such as "sofa," "blue," and "modern" into the interface. The terminal converts these requests into an appropriate format (e.g., JSON format) and sends them to the server.

[1007] Next, the server receives the request data sent by the user. The server analyzes this data and processes it as input data for the generative AI model and emotion engine. The generative AI model generates a 3D model and texture for the item based on this input data. The emotion engine also recognizes the user's emotion and adjusts the parameters of the generative AI model to generate an item that best suits the user's emotion. The generated item data is temporarily stored on the server.

[1008] The generated item data is sent from the server to the device for user confirmation. The user can check the generated results on the device and input correction instructions as necessary. For example, if the color of a sofa differs from what the user's emotions were considered, the emotion engine analyzes the user's emotions from their facial expressions, voice, and text input and suggests appropriate corrections. If the user accepts the suggestions, the device reformats the correction instructions in JSON format and sends them to the server.

[1009] The server uses the generative AI model based on new instructions and the user's emotional data to generate a revised item. This allows the user to repeatedly make revisions until they achieve the desired result. Finally, if the user is satisfied with the generated item, the server registers this item data in the metaverse space. The registered item can then be placed and used by the user within the metaverse space. The user can then adjust the position and orientation of the item within their metaverse space to complete the decoration of their room.

[1010] Specific examples

[1011] The specific flow when a user creates a "blue modern sofa" is shown below.

[1012] 1. Request Input

[1013] The user types "sofa," "blue," and "modern" into the device interface.

[1014] The terminal converts this into JSON format and sends it to the server.

[1015] 2. Processing of request data and use of generation AI models and emotion engines

[1016] The server receives and analyzes the request data.

[1017] The generative AI model is given the inputs "sofa," "blue," and "modern," and generates a 3D model and texture of a blue modern sofa.

[1018] The emotion engine analyzes the user's facial expressions and voice and adjusts the parameters of the generative AI model.

[1019] 3. Providing and Modifying Generated Results

[1020] The server transmits the generated sofa data to the terminal.

[1021] The user checks the result and inputs a correction request such as "make it a little brighter blue."

[1022] The emotion engine analyzes the user's emotions and suggests the most suitable colors.

[1023] The server inputs the modification request back into the generative AI model and generates the modified sofa.

[1024] 4. Registration and placement in the metaverse space

[1025] Finally, the server registers the sofa that the user is satisfied with in the metaverse space.

[1026] The user places the sofa in the metaverse space and completes the decoration.

[1027] This system allows users to easily generate and arrange high-quality content without specialized knowledge, and also provides optimal items that take into account the user's emotions.

[1028] The processing flow will be explained below.

[1029] Step 1:

[1030] The user enters a request

[1031] The user uses the device interface to input a request for the item they want to create. For example, to specify a "blue modern sofa," they input "sofa," "blue," and "modern." The device then converts this request into JSON-formatted data.

[1032] Step 2:

[1033] The device sends the requested data to the server

[1034] The terminal sends the converted request data to the server as an HTTP request. The server analyzes the received request data and converts it into a format that can be handled internally.

[1035] Step 3:

[1036] The server inputs data into the generated AI model and emotion engine

[1037] The server passes the analyzed request data to the generative AI model and emotion engine as input data. The generative AI model generates a 3D model and texture of the requested item based on the input data. Meanwhile, the emotion engine collects and analyzes data such as the user's facial expressions, voice, and text input to analyze the user's emotions.

[1038] Step 4:

[1039] Generative AI model and emotion engine generate items and emotion data

[1040] The generative AI model automatically generates a 3D model and texture of a blue modern sofa based on the specifications of "sofa," "blue," and "modern." Meanwhile, the emotion engine analyzes the user's input data and real-time emotion data to suggest the optimal items to satisfy the user. The generated item data and emotion data are returned to the server.

[1041] Step 5:

[1042] The server returns the generated results to the terminal

[1043] The server organizes the generated item data and the analysis results of the emotion engine and sends them back to the device, which then displays the generated results and suggestions to the user.

[1044] Step 6:

[1045] The user can review the generated results and suggestions and enter correction requests if necessary.

[1046] The user checks the generated items and the emotion engine's suggestions on the device. For example, if the color is different from what they expected, the user can input a correction request into the device, such as "make it a lighter blue." The device then converts this into JSON format and sends it to the server.

[1047] Step 7:

[1048] The server generates a correction request again and inputs it into the AI ​​model and emotion engine.

[1049] The server receives the user's modification request and again inputs the data into the generative AI model and emotion engine. The generative AI model generates a modified 3D model and texture based on the new input data, and the emotion engine generates new emotion data and suggestions. The generated data is returned to the server and sent back to the device.

[1050] Step 8:

[1051] The user confirms the item and confirms it

[1052] The user reviews the results again and repeats steps 6 and 7 as necessary until satisfied, finally finalizing the satisfactory items.

[1053] Step 9:

[1054] The server registers the item data in the metaverse space.

[1055] The server registers the confirmed item data in the metaverse space database. The item is reflected in the user's metaverse space.

[1056] Step 10:

[1057] Users place items in the metaverse space

[1058] The user logs in to the metaverse space using a terminal and places the generated items wherever they like. The user adjusts the position and orientation of the items to complete the final layout.

[1059] Example 2

[1060] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1061] In conventional systems, it was difficult for users to generate specific items while taking their emotions into consideration and providing optimal items. Furthermore, reflecting requests for revisions to generated items required manual work and multiple trial and error processes, which was inefficient. There was also a need for a system that could centrally manage these processes and be intuitive for users to operate.

[1062] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1063] In this invention, the server includes means for processing requests received from users and generating input data for generation, means for automatically generating items based on the input data using a generative model, means for providing the generated item data to the user and generating the item again in response to a request for confirmation and correction, means for recognizing the user's emotions and adjusting the parameters of the generative model in accordance with the emotions, and means for registering and arranging the finalized item data in a virtual space. This allows users to efficiently generate and modify optimal items that reflect their emotions and arrange them in a virtual space.

[1064] "User" refers to a person who utilizes a terminal to input requests into the system and to review and modify generated items.

[1065] "Server" refers to a computer system that receives request data from a user and generates and modifies items using generative models and emotion recognition engines.

[1066] A "terminal" refers to a device operated by a user to input requests and communicate with a server. Specifically, this applies to smartphones and computers.

[1067] "Request data" refers to data obtained by converting detailed information about an item that a user inputs through a terminal into a different format.

[1068] A "generative model" refers to an artificial intelligence model that automatically generates 3D models and textures based on required data.

[1069] An "emotion recognition engine" refers to a software system that analyzes a user's facial expressions and voice to identify the user's emotions.

[1070] "Virtual space" refers to a three-dimensional computer-generated environment constructed using digital technology that users can manipulate and experience as if it were a physical space through an interface.

[1071] "Input data" refers to data used by the generative model based on user request data.

[1072] The system of the present invention automatically generates items in a virtual space based on user requests, recognizes the user's emotions, and suggests or modifies items according to the emotions. This system includes a series of processes: request input, request data processing, use of a generation AI model and an emotion recognition engine, provision of the generated results and reflection of modification requests, and final registration and placement of the item data in the virtual space.

[1073] First, a user uses a terminal to input a request for creating an item. For example, if a user wants to create a "blue modern sofa," the user inputs detailed instructions such as "sofa," "blue," and "modern" into the interface. The terminal converts these requests into an appropriate format (e.g., JSON format) and sends them to the server.

[1074] The server receives and analyzes the request data sent by the user. The analyzed data is passed to the generative AI model and emotion recognition engine. The generative AI model generates a 3D model and texture of the item based on this input data. Meanwhile, the emotion recognition engine recognizes the user's emotion and adjusts the parameters of the generative AI model. For example, if the user is smiling, the emotion recognition engine instructs the generated item to use bright colors. The generated item data is temporarily stored on the server.

[1075] The generated item data is sent from the server to the device for user confirmation. The user can review the generated results on the device and input correction requests as needed. For example, if the color of the sofa is darker than expected, the user can input a correction request such as "make it a lighter blue." The emotion recognition engine analyzes the user's emotions and the new request and suggests the optimal correction. The server then uses the generative AI model again based on the new instructions to generate the corrected item.

[1076] Finally, if the user is satisfied with the generated item, the server registers the item data in the virtual space. The registered item can then be placed and used by the user in the virtual space. The user can then adjust the position and orientation of the item in their virtual space to complete the room decoration.

[1077] Specific examples

[1078] A specific example prompt for a user to generate a "blue modern sofa" would be:

[1079] "Generate a blue modern sofa."

[1080] "Please suggest a blue sofa that reflects the emotion you read from the user's facial expression."

[1081] This system allows users to easily create and arrange high-quality items without specialized knowledge, and also provides optimal items that take into account the user's emotions.

[1082] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1083] Step 1: User enters item creation request

[1084] The user inputs details of the item they want to create (e.g., "sofa," "blue," "modern") into the device interface. The device receives this as input data and converts it into JSON format using a format conversion tool. The input is the user's instructions, and the output is JSON format data.

[1085] Step 2: The terminal converts the requested data into a different format and sends it

[1086] The terminal converts the item creation request entered by the user into JSON format and then sends this data to the server. Here, the input is the detailed request data from the user, and the output is the JSON data sent to the server.

[1087] Step 3: Parsing and processing the request data by the server

[1088] The server receives the JSON data sent from the device and analyzes it. The analyzed content includes the item type, color, style, etc. The analyzed data is passed as input to the generative AI model and emotion recognition engine. The input is JSON data, and the output is the analyzed item information.

[1089] Step 4: Generate items using a generative AI model

[1090] The server passes the analyzed item information to the generative AI model, which generates a 3D model and texture according to the specified conditions (e.g., "blue modern sofa"). The generated data is temporarily stored on the server. The input is the analyzed item information, and the output is the generated 3D model and texture.

[1091] Step 5: Recognize and adjust the user's emotions using the emotion recognition engine

[1092] The server uses an emotion recognition engine to recognize the user's emotions. This involves analyzing the user's facial expressions and voice using the device's camera and microphone. The input is the user's facial expression and voice data, and the output is the recognized emotional information. This emotional information is used to adjust the parameters of the generative AI model and reflect it in the output.

[1093] Step 6: Server saves and sends the generated results

[1094] The server temporarily stores the generated and adjusted item data and then transmits it to the user's device. The input is the adjusted item data, and the output is the data sent to the device.

[1095] Step 7: User checks the generated results and requests corrections

[1096] The user checks the generated item on the terminal interface and inputs correction requests as needed. For example, the user may request "make it a lighter blue." The input is the generated item data, and the output is the user's correction requests.

[1097] Step 8: Emotion Recognition Engine Suggests Corrections

[1098] The emotion recognition engine proposes optimal corrections based on the user's new request and the results of facial and voice analysis. The input is the user's new request and emotional data, and the output is the optimal correction proposal.

[1099] Step 9: Server creates and resubmits the corrected item

[1100] The server uses the generative AI model again based on the user's correction request to generate a corrected item. This corrected item is then resent to the user's device. The input is the correction instruction and emotion data, and the output is the corrected item data.

[1101] Step 10: Register and place items in the metaverse space

[1102] The server registers the items that the user is finally satisfied with in the virtual space. After registration is complete, the user can freely place items in the virtual space and customize the environment. The input is the finalized item data, and the output is the items that have been registered in the virtual space and can now be placed.

[1103] (Application example 2)

[1104] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1105] Previously, when creating items in a virtual store or metaverse space, users had to manually input and edit data using a limited interface, which was extremely time-consuming and laborious. Furthermore, if the created item did not match the user's emotions or preferences, the user would have to re-edit it, resulting in a poor user experience. The present invention aims to solve these problems and provide a system that allows users to create and edit items more comfortably and efficiently.

[1106] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing the user's emotions in real time and adjusting the parameters of the generation AI model based on the emotion data, means for providing the generated item data to the user and generating items again in response to confirmation and correction requests, and means for registering and arranging the finalized item data in the virtual space. This allows the user to easily generate and modify optimal items based on their emotions and preferences, and efficiently arrange them.

[1107] "Means for recognizing a user's emotions in real time and adjusting the parameters of a generative AI model based on the emotion data" refers to a system that has the ability to recognize emotions in real time not only from data entered by the user but also from the user's facial expressions, voice, etc., and dynamically adjust the operating parameters of a generative AI model according to those emotions.

[1108] A "generative AI model" is an artificial intelligence model that automatically generates specific items based on input data, including processes that generate 3D models and textures.

[1109] The "emotion engine" analyzes the user's emotions and suggests or modifies items based on that emotion data, thereby providing the optimal items that the user desires.

[1110] "Virtual space" refers to a virtual area where users can engage in various activities within a digital space, such as a metaverse environment or a virtual store provided via the Internet.

[1111] The "means for generating an item again in response to a confirmation and correction request" is a system function that, when a user confirms a generated item and requests correction as necessary, generates the item again based on that request.

[1112] "Means for registering and arranging finalized item data in a virtual space" is a function that allows a user to register items that have been confirmed to their satisfaction in a virtual space and arrange them freely within that virtual space.

[1113] The "format conversion means for converting a user's request into input data" is a means having the function of appropriately formatting a request input by a user and converting it into a form that is easily understood by the system.

[1114] The system according to the present invention automatically generates items in a virtual space based on requests from a user, and further recognizes the user's emotions and suggests or modifies items according to the emotions.

[1115] System Program

[1116] Basic configuration

[1117] User terminal: A device used by users to input requests and check generated items. This can be a smart glass or a smartphone.

[1118] Server: The central device that processes request data and generates items using generative AI models and emotion engines.

[1119] Generative AI model: An artificial intelligence model that generates items based on user requests.

[1120] Emotion engine: An engine that analyzes user emotion data and adjusts the parameters of the generative AI model based on that data.

[1121] Explanation of program processing

[1122] The server first receives a request entered by the user via their device. This request is converted into a data format such as JSON. The server then inputs the converted data into a generative AI model to generate a 3D model and texture of the item based on the request. The emotion engine also recognizes emotional data from the user's facial expressions, voice, and text input, and adjusts the parameters of the generative AI model.

[1123] The generated item is immediately sent to the user's terminal, where the user can confirm it. If the user requires any modifications, the request is sent again to the server. This allows the optimal item to be generated according to the user's feelings. Modifications continue until the user is satisfied, and the finalized item is registered in the virtual space.

[1124] Specific examples

[1125] Suppose a user accesses a virtual store using smart glasses and enters a request by voice, such as "a blue modern sofa." The keywords "sofa," "blue," and "modern" are transmitted to the server through the interface of the smart glasses.

[1126] The server receives this and generates a 3D model and texture of a "blue modern sofa." At the same time, the emotion engine detects a "relaxed atmosphere" from the user's smile and adjusts the color of the sofa slightly brighter accordingly. The generated result is presented to the user, and if the user requests a correction such as "a darker blue," the item is generated again using the same process.

[1127] Example prompt sentence:

[1128] "Create a modern blue sofa that has a relaxing atmosphere."

[1129] This system allows users to easily create items that best suit their emotions and preferences, and efficiently place them in the virtual space.

[1130] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1131] Step 1:

[1132] A user uses a device (such as smart glasses or a smartphone) to input requests such as "sofa," "blue," and "modern" into the user interface. This input request data is converted into a data format such as JSON and sent to the server.

[1133] Input: User request (e.g. "sofa", "blue", "modern")

[1134] Output: Formatted request data (e.g., JSON format)

[1135] Specific operation:

[1136] When a user inputs something using voice or gestures, the device analyzes it and converts it into text data, which it then converts into JSON format and sends to the server via the network.

[1137] Step 2:

[1138] The server receives and analyzes the request data sent from the device. The analyzed data is input into the generative AI model to generate a 3D model and texture of a "blue modern sofa." The generated temporal data is also adjusted by the emotion engine based on the user's emotional data.

[1139] Input: Formatted request data

[1140] Output: Generated item data (3D models and textures)

[1141] Specific operation:

[1142] The server analyzes the received JSON data and passes the keywords "sofa," "blue," and "modern" to the generative AI model. The generative AI model creates a 3D model and texture based on the data. At the same time, the emotion engine analyzes the user's emotion data and adjusts the parameters of the generative AI model to generate the optimal output.

[1143] Step 3:

[1144] The generated item data is temporarily stored and sent to the user's terminal. The user can check the generated item on the terminal and, if not satisfied, input a correction request. This correction request is sent back to the server.

[1145] Input: Generated item data

[1146] Output: Correction requests from the user

[1147] Specific operation:

[1148] The item data is sent from the server to the device, where the user confirms it. If the user wishes to make any changes, they can enter their request by voice or text, such as "make it a lighter blue." The device then converts this request back into JSON format and sends it to the server.

[1149] Step 4:

[1150] The server receives the modification request from the user again and generates the item again using the generative AI model and emotion engine, thereby providing an item optimized for the user's emotions.

[1151] Input: User request for correction (e.g., "Make it a lighter blue")

[1152] Output: Regenerated item data

[1153] Specific operation:

[1154] The server receives the correction request, inputs it back into the generative AI model as data, and uses the emotion engine to readjust it based on the user's emotions. The generative AI model then creates new 3D models and textures and sends the data to the user's device.

[1155] Step 5:

[1156] Finally, once an item that satisfies the user is created, the server registers and places the item data in the virtual space. The user can then adjust the position and orientation of the item within the virtual space to complete the decoration of the space.

[1157] Input: Finalized item data

[1158] Output: Items registered in the virtual space

[1159] Specific operation:

[1160] The server receives the item data that the user is ultimately satisfied with and registers it in the specified location within the virtual space. The user can then freely adjust the position and orientation of the items within the virtual space via their device to complete the decoration of the space.

[1161] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1162] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1163] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1164] [Fourth embodiment]

[1165] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1166] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1167] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1168] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1169] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1170] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1171] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1172] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1173] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1174] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1175] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1176] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1177] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1178] The system of the present invention automatically generates items in the Metaverse space based on user requests. This system includes a series of processes: the user inputs the request using a terminal, the server processes the request data and uses a generation AI model, the server provides the generated results and reflects correction instructions, and the final item data is registered and placed in the Metaverse space.

[1179] First, a user uses a terminal to input a request for creating an item. For example, if a user wants to create a "blue modern sofa," the user inputs detailed instructions such as "sofa," "blue," and "modern" into the interface. The terminal converts these requests into an appropriate format (e.g., JSON format) and sends them to the server.

[1180] Next, the server receives the request data sent by the user. The server analyzes this data and processes it as input data for the generative AI model. The generative AI model generates a 3D model and texture of the item based on this input data. The generated item data is temporarily stored on the server.

[1181] The generated item data is sent from the server to the device for confirmation by the user. The user can check the generated results provided on the device and input correction instructions as necessary. For example, if the user wants to change the color of the sofa to a "lighter blue," they can send that instruction again from the device to the server. The server will then use the generative AI model again based on the new instructions to generate the corrected item. This allows the user to make repeated corrections until they achieve the desired result.

[1182] Finally, if the user is satisfied with the created item, the server registers this item data in the metaverse space. The registered item can then be placed and used by the user in the metaverse space. The user can then adjust the position and orientation of the item in their metaverse space to complete the room decoration.

[1183] Specific examples

[1184] The specific flow when a user creates a "blue modern sofa" is shown below.

[1185] 1. Request Input

[1186] The user types "sofa," "blue," and "modern" into the device interface.

[1187] The terminal converts this into JSON format and sends it to the server.

[1188] 2. Processing requirements data and using generative AI models

[1189] The server receives and analyzes the request data.

[1190] The generative AI model is given the inputs "sofa," "blue," and "modern," and generates a 3D model and texture of a blue modern sofa.

[1191] 3. Providing and Modifying Generated Results

[1192] The server transmits the generated sofa data to the terminal.

[1193] The user checks the result and inputs a correction request such as "make it a little brighter blue."

[1194] The server inputs the modification request back into the generative AI model and generates the modified sofa.

[1195] 4. Registration and placement in the metaverse space

[1196] Finally, the server registers the sofa that the user is satisfied with in the metaverse space.

[1197] The user places the sofa in the metaverse space and completes the decoration.

[1198] The system allows users to easily generate and distribute high-quality content without specialized knowledge.

[1199] The processing flow will be explained below.

[1200] Step 1:

[1201] The user enters a request

[1202] The user uses the device interface to input a request for the item they want to create. For example, to specify a "blue modern sofa," they input "sofa," "blue," and "modern." The device then converts this request into JSON-formatted data.

[1203] Step 2:

[1204] The device sends the requested data to the server

[1205] The terminal sends the converted request data to the server as an HTTP request. The server analyzes the received request data and converts it into a format that can be handled internally.

[1206] Step 3:

[1207] The server inputs data into the generative AI model

[1208] The server passes the analyzed request data as input to the generative AI model, which then generates a 3D model and texture for the requested item based on the input data.

[1209] Step 4:

[1210] Generative AI model generates items

[1211] The generative AI model automatically generates a 3D model and texture of a blue modern sofa based on the specifications of "sofa," "blue," and "modern." The generated item data is then returned to the server.

[1212] Step 5:

[1213] The server returns the generated results to the terminal

[1214] The server organizes the item data returned by the generative AI model and sends it back to the device, which then displays the generated results to the user.

[1215] Step 6:

[1216] The user checks the generated results and inputs correction requests as necessary.

[1217] The user checks the generated item on the device. For example, if the color is different from what they expected, they input a correction request into the device, such as "make it a lighter blue." The device then reformats this into JSON and sends it to the server.

[1218] Step 7:

[1219] The server inputs the correction request into the generated AI model again.

[1220] The server receives the user's modification request and inputs the data back into the generative AI model, which then generates a new item based on the modification request.

[1221] Step 8:

[1222] The generative AI model generates the modified item.

[1223] The generative AI model generates a modified 3D model and texture based on the new input data and returns it to the server, which then sends the modified item data back to the device.

[1224] Step 9:

[1225] The user confirms the item and confirms it

[1226] The user reviews the results again and repeats steps 6 through 8 as necessary until satisfied, finally finalizing the items they are satisfied with.

[1227] Step 10:

[1228] The server registers the item data in the metaverse space.

[1229] The server registers the confirmed item data in the metaverse space database. The item is reflected in the user's metaverse space.

[1230] Step 11:

[1231] Users place items in the metaverse space

[1232] The user logs in to the metaverse space using a terminal and places the generated items wherever they like. The user adjusts the position and orientation of the items to complete the final layout.

[1233] Example 1

[1234] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1235] In the Metaverse, it has been difficult for users to easily generate high-quality items and place them as desired. Conventional systems require advanced expertise and complex operations, placing a heavy burden on many users. In addition, there is also the problem that if a generated item does not completely meet the user's requirements, it is not easy to regenerate or modify it.

[1236] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1237] In this invention, the server includes means for processing requests received from users and generating input data for generating items, means for converting the user's request into JSON format using a terminal and transmitting the JSON data to the server, means for analyzing the JSON data received by the server and formatting it as input data for the generative AI model, means for automatically generating items based on the input data using the generative AI model, means for temporarily storing the generated item data on the server, means for providing the generated item data to the user and generating items again in response to the user's confirmation and correction requests, means for registering and arranging item data that the user is satisfied with in the metaverse space, and means for adjusting the position and orientation of registered items in the metaverse space. This enables users without specialized knowledge to easily generate high-quality items, make corrections until they meet their requirements, and finally place them in the metaverse space.

[1238] "User" refers to an individual or organization that uses this system to generate and place items.

[1239] "Terminal" refers to a computer or mobile device used by a User to register input data and view generated items.

[1240] "Server" refers to a computer system that receives user requests, analyzes the request data, generates items using a generative AI model, and registers the item data in the Metaverse space.

[1241] "Means for generating input data" refers to the function of converting a request received from a user into the data format (e.g., JSON format) required for item generation.

[1242] "JSON format" is an abbreviation for JavaScript Object Notation, and refers to a lightweight data exchange format for expressing data in a format that is easy for humans to read and machines to parse.

[1243] "Generative AI model" refers to an artificial intelligence model that automatically generates items such as 3D models and textures based on input data.

[1244] "Item Data" refers to digital data, including 3D models and textures, generated by a generative AI model.

[1245] The "metaverse space" is a virtual digital world, an online environment where users can place and interact with registered items.

[1246] "Means for adjusting position and orientation" refers to the function that allows users to change the position and orientation of registered items in the metaverse space and optimize their placement.

[1247] "Means for generating an item again in response to confirmation and correction requests" refers to the function that, when the user confirms the generated results provided and inputs a correction request, regenerates the item using the generation AI model again based on that request.

[1248] The system of the present invention automatically generates items in the Metaverse space based on user requests. This system includes a series of processes: the user inputs the request using a terminal, the server processes the request data and uses a generation AI model, the server provides the generated results and reflects correction instructions, and the final item data is registered and placed in the Metaverse space.

[1249] First, a user uses a terminal to input a request for creating an item. For example, if a user wants to create a "blue modern sofa," the user inputs detailed instructions such as "sofa," "blue," and "modern" into the interface. The terminal converts these requests into an appropriate format (e.g., JSON format) and sends them to the server.

[1250] Next, the server receives the request data sent by the user. The server analyzes this data and formats it as input data for the generative AI model. The generative AI model generates a 3D model and texture of the item based on this input data. The generated item data is temporarily stored on the server.

[1251] The generated item data is sent from the server to the device for confirmation by the user. The user can check the generated results provided on the device and input correction instructions as necessary. For example, if the user wants to change the color of the sofa to a "lighter blue," they can send that instruction again from the device to the server. The server will then use the generative AI model again based on the new instructions to generate the corrected item. This allows the user to make repeated corrections until they achieve the desired result.

[1252] Finally, if the user is satisfied with the created item, the server registers this item data in the metaverse space. The registered item can then be placed and used by the user in the metaverse space. The user can then adjust the position and orientation of the item in their metaverse space to complete the room decoration.

[1253] Hardware and software used

[1254] Server: A high-performance server (e.g., a server equipped with an NVIDIA GPU)

[1255] Generative AI model: Examples of models include OpenAI's DALL-E and Google's DeepMind

[1256] Data format: JSON

[1257] Specific examples

[1258] The specific flow when a user creates a "blue modern sofa" is shown below.

[1259] 1. Request Input

[1260] The user types "sofa," "blue," and "modern" into the device interface.

[1261] The terminal converts this into JSON format and sends it to the server.

[1262] 2. Processing requirements data and using generative AI models

[1263] The server receives and analyzes the request data.

[1264] The generative AI model is given the inputs "sofa," "blue," and "modern," and generates a 3D model and texture of a blue modern sofa.

[1265] 3. Providing and Modifying Generated Results

[1266] The server transmits the generated sofa data to the terminal.

[1267] The user checks the result and inputs a correction request such as "make it a little brighter blue."

[1268] The server inputs the modification request back into the generative AI model and generates the modified sofa.

[1269] 4. Registration and placement in the metaverse space

[1270] Finally, the server registers the sofa that the user is satisfied with in the metaverse space.

[1271] The user places the sofa in the metaverse space and completes the decoration.

[1272] Prompt Sentence Examples

[1273] Here are some example input prompts for a generative AI model:

[1274] Example prompt

[1275] "Sofa" "Blue" "Modern"

[1276] The system allows users to easily generate and distribute high-quality content without requiring specialized knowledge.

[1277] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1278] Step 1:

[1279] The user inputs a request for item creation into the terminal interface. The user inputs a specific request, such as "sofa," "blue," and "modern," into the terminal. This causes the terminal to prepare the data for step 2.

[1280] Input: User request (e.g. "sofa", "blue", "modern")

[1281] Output: Request to convert input data to JSON format

[1282] Specific behavior:

[1283] The user enters a keyword into the input field on the device.

[1284] The terminal receives input data.

[1285] Step 2:

[1286] The terminal converts the user's request into JSON format. The converted JSON data is sent to the server. At this stage, the data format is unified and formatted so that it can be passed to the server.

[1287] Input: User request data

[1288] Output: Data converted to JSON format

[1289] Specific behavior:

[1290] The terminal converts the input data into JSON format.

[1291] The terminal sends the converted JSON data to the server as an HTTP request.

[1292] Step 3:

[1293] The server parses the received JSON data and formats it as input data for the generative AI model. The JSON data is parsed and converted into a prompt that the AI ​​model can understand.

[1294] Input: User request data in JSON format

[1295] Output: Input data to the generative AI model

[1296] Specific behavior:

[1297] The server deserializes and parses the JSON data.

[1298] The server formats the data as required.

[1299] Step 4:

[1300] The server inputs the formatted data into a generative AI model to generate a 3D model and texture for the item. The generative AI model generates the item based on the input data, and the data is temporarily stored on the server.

[1301] Input: Formatted input data

[1302] Output: 3D model and texture data of the generated item

[1303] Specific behavior:

[1304] The server sends the formatted data to the generative AI model.

[1305] Generative AI model generates 3D models and textures.

[1306] The server temporarily stores the generated data.

[1307] Step 5:

[1308] The server sends the generated item data to the terminal and receives confirmation from the user. The user confirms the provided generation results on the terminal and proceeds to the next step.

[1309] Input: Generated item data

[1310] Output: Generated data provided to the user

[1311] Specific behavior:

[1312] The server sends the generated data to the terminal as an HTTP response.

[1313] The device displays the provided data.

[1314] Step 6:

[1315] The user checks the generated results and inputs correction instructions as needed. For example, they may request a correction such as "make it a lighter blue." The device then converts this new instruction back into JSON format and sends it to the server.

[1316] Input: User correction instructions

[1317] Output: Request to convert correction instructions to JSON format

[1318] Specific behavior:

[1319] The user enters correction instructions into the interface.

[1320] The device converts the correction instructions into JSON format and sends them to the server.

[1321] Step 7:

[1322] The server receives the modification instructions, inputs them into the generation AI model again, and regenerates the item. The item is modified based on the new instructions.

[1323] Input: Correction instruction data

[1324] Output: 3D model and texture data of the modified item

[1325] Specific behavior:

[1326] The server analyzes the correction instructions and re-inputs them into the generative AI model.

[1327] A generative AI model generates modified items.

[1328] The server saves the modified data.

[1329] Step 8:

[1330] If the user is satisfied with the final generated item, the server registers this item data in the metaverse space.

[1331] Input: Satisfied item data

[1332] Output: Registration data in the metaverse space

[1333] Specific behavior:

[1334] The user makes a final confirmation and indicates satisfaction.

[1335] The server registers the item data in the metaverse space.

[1336] Step 9:

[1337] Users can complete the decoration of a room by placing registered items in the metaverse space and adjusting their position and orientation.

[1338] Input: Registered item data

[1339] Output: Position and orientation of placed items

[1340] Specific behavior:

[1341] A user logs into the metaverse space.

[1342] The user arranges the items and adjusts their position and orientation.

[1343] Through this series of steps, users can easily create high-quality items without any specialized knowledge, repeatedly modify them until they meet their requirements, and finally place them in the Metaverse space.

[1344] (Application example 1)

[1345] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1346] Conventional methods for building metaverses and virtual stores require users to have specialized design knowledge, making them difficult for average users. Customizing and arranging items also requires a lot of time and effort, making efficient design difficult. Therefore, there was a demand for a system that would allow users to easily design high-quality virtual stores.

[1347] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1348] In this invention, the server includes means for processing requests received from users and generating input data for generating items, means for automatically generating items based on the input data using a generative AI model, means for providing the generated item data to the user and generating items again in response to confirmation and correction requests, means for registering and arranging the finalized item data in the metaverse space, and means for providing a visual interface for arranging and customizing the generated items in a virtual store. This enables users, even without specialized knowledge, to easily and efficiently design and customize high-quality virtual stores.

[1349] "Means for processing requests received from users and generating input data for item generation" is a function that analyzes requests entered by users and converts them into data in a format suitable for item generation.

[1350] "Means for automatically generating items based on the input data using a generative AI model" refers to a function that passes input data to an AI model and automatically generates items such as 3D models and textures based on that data.

[1351] "Means for providing the generated item data to the user and generating the item again in response to confirmation and correction requests" is a function that shows the generated item data to the user and, if necessary, generates the item again by reflecting correction requests made by the user.

[1352] The "means for registering and arranging the finalized item data in the metaverse space" is a function for registering the item data that the user has finally confirmed in the metaverse space and arranging it.

[1353] "Means for providing a visual interface for arranging and customizing items generated in a virtual store" is a function that provides a user with an interface that allows them to visually arrange and customize items generated in a virtual store.

[1354] The "format conversion means" is a means for converting a user request into input data.

[1355] The "process of generating a 3D model and texture" refers to the procedure by which an AI model generates a three-dimensional shape and its surface material based on a user's request.

[1356] A system for implementing the present invention automatically generates items for a virtual store based on user requests. This system includes a series of processes: a user inputs the request using a terminal, a server processes the request data and uses a generation AI model, a server provides the generated results and reflects correction instructions, and finally, the system registers and places the final item data in the virtual store.

[1357] First, a user uses a terminal to input a request for creating an item. For example, if a user wants to create a "wooden shelf" or a "retro lamp," the user inputs these requests through the interface. The terminal converts these requests into an appropriate format (e.g., JSON format) and sends them to the server.

[1358] Next, the server receives the request data sent by the user. The server analyzes this data and processes it as input data for the generative AI model. The generative AI model used by the server is generally composed of AI technologies such as GPT-4 and Stable Diffusion. The generative AI model generates a 3D model and texture of the item based on the input data. The generated item data is temporarily stored on the server.

[1359] The generated item data is sent from the server to the device for confirmation by the user. The user can check the generated results on the device and input correction instructions as needed. For example, a request for a "sofa with a lighter blue color" is possible. The server then uses the generative AI model again based on the new instructions to generate the corrected item. This allows the user to repeatedly make corrections until they achieve the desired result.

[1360] Finally, if the user is satisfied with the generated items, the server registers the item data in the virtual store space. The registered items become available for the user to place and use in the virtual store. The user adjusts the position and orientation of the items in their virtual store to complete the design of the store interior.

[1361] To give a specific example, when a user wants to generate a "wooden shelf," the following steps are taken: First, the user enters "wooden shelf" into the device interface. The device then converts this into JSON format and sends it to the server. The server receives the request data, passes "wooden shelf" as input to the generative AI model, and generates a 3D model and texture. The generated shelf data is sent to the device for confirmation by the user. If necessary, a correction request can be made, such as "make it a lighter color," and the corrected data is then generated again.

[1362] The above system configuration enables users to easily and efficiently design and customize high-quality virtual stores, even without specialized knowledge.

[1363] Example prompt sentences to use:

[1364] "Create a modern, high-quality blue sofa."

[1365] "Please change it to a lighter blue."

[1366] "Generate a wooden shelf."

[1367] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1368] Step 1:

[1369] The user inputs a request to create an item through the terminal interface. The user inputs a detailed prompt such as "wooden shelf" or "retro lamp." The input prompt is converted into an appropriate format (e.g., JSON format) by the terminal. Here, the user's request is the input, and the converted formatted data is the output.

[1370] Step 2:

[1371] The terminal transmits the format-converted data to the server. Specifically, the terminal uploads the converted data to the server via the network. Here, the format-converted data is the input, and the requested data received by the server is the output.

[1372] Step 3:

[1373] The server analyzes the received request data and creates input data for the generative AI model. Specifically, the server analyzes the JSON format data and formats it in a format that the generative AI model can understand. Here, the received request data is received as input, and the input data for the generative AI model is obtained as output.

[1374] Step 4:

[1375] The server uses a generative AI model to generate a 3D model and texture for the item. Specifically, the server passes input data to a generative AI model (such as GPT-4 or Stable Diffusion) and generates the item based on that data. Here, the input to the generative AI model is the input, and the generated 3D model and texture data are obtained as the output.

[1376] Step 5:

[1377] The server sends the generated 3D model and texture data to the terminal and provides it to the user. Specifically, the server sends the generated results to the terminal via the network. Here, the generated 3D model and texture data are the input, and the generated results displayed on the terminal are obtained as the output.

[1378] Step 6:

[1379] The user uses the terminal to check the generated results and input correction requests as necessary. The user may input a correction request such as "make the color a little brighter" into the terminal. Here, the generated item data and the user's correction request are input, and the correction request is sent from the terminal to the server as output.

[1380] Step 7:

[1381] The server receives the modification request and generates the modified item using the generative AI model again. Specifically, the server analyzes the modification request and passes it to the generative AI model again to generate a new 3D model and texture. Here, the modification request is the input and the modified 3D model and texture data are obtained as the output.

[1382] Step 8:

[1383] Finally, if the user is satisfied with the generated item, the server registers this item data in the virtual store space. Specifically, the server saves the item data in storage so that it can be placed in the user's virtual store. Here, the input is the finalized item data, and the output is the item registered in the virtual store.

[1384] Step 9:

[1385] Users use their devices to arrange and customize items in a virtual store. Specifically, they adjust the position and orientation of items using a visual interface. Here, the input is the items displayed in the virtual store, and the output is the final store design.

[1386] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1387] The system of the present invention not only automatically generates items in the metaverse space based on user requests, but also recognizes the user's emotions and suggests or modifies items according to those emotions. This system involves a series of processes: the user inputs the request using a terminal, the server processes the request data and uses a generation AI model and emotion engine, the server provides the generated results and reflects modification instructions, and the final item data is registered and placed in the metaverse space.

[1388] First, a user uses a terminal to input a request for creating an item. For example, if a user wants to create a "blue modern sofa," the user inputs detailed instructions such as "sofa," "blue," and "modern" into the interface. The terminal converts these requests into an appropriate format (e.g., JSON format) and sends them to the server.

[1389] Next, the server receives the request data sent by the user. The server analyzes this data and processes it as input data for the generative AI model and emotion engine. The generative AI model generates a 3D model and texture for the item based on this input data. The emotion engine also recognizes the user's emotion and adjusts the parameters of the generative AI model to generate an item that best suits the user's emotion. The generated item data is temporarily stored on the server.

[1390] The generated item data is sent from the server to the device for user confirmation. The user can check the generated results on the device and input correction instructions as necessary. For example, if the color of a sofa differs from what the user's emotions were considered, the emotion engine analyzes the user's emotions from their facial expressions, voice, and text input and suggests appropriate corrections. If the user accepts the suggestions, the device reformats the correction instructions in JSON format and sends them to the server.

[1391] The server uses the generative AI model based on new instructions and the user's emotional data to generate a revised item. This allows the user to repeatedly make revisions until they achieve the desired result. Finally, if the user is satisfied with the generated item, the server registers this item data in the metaverse space. The registered item can then be placed and used by the user within the metaverse space. The user can then adjust the position and orientation of the item within their metaverse space to complete the decoration of their room.

[1392] Specific examples

[1393] The specific flow when a user creates a "blue modern sofa" is shown below.

[1394] 1. Request Input

[1395] The user types "sofa," "blue," and "modern" into the device interface.

[1396] The terminal converts this into JSON format and sends it to the server.

[1397] 2. Processing of request data and use of generation AI models and emotion engines

[1398] The server receives and analyzes the request data.

[1399] The generative AI model is given the inputs "sofa," "blue," and "modern," and generates a 3D model and texture of a blue modern sofa.

[1400] The emotion engine analyzes the user's facial expressions and voice and adjusts the parameters of the generative AI model.

[1401] 3. Providing and Modifying Generated Results

[1402] The server transmits the generated sofa data to the terminal.

[1403] The user checks the result and inputs a correction request such as "make it a little brighter blue."

[1404] The emotion engine analyzes the user's emotions and suggests the most suitable colors.

[1405] The server inputs the modification request back into the generative AI model and generates the modified sofa.

[1406] 4. Registration and placement in the metaverse space

[1407] Finally, the server registers the sofa that the user is satisfied with in the metaverse space.

[1408] The user places the sofa in the metaverse space and completes the decoration.

[1409] This system allows users to easily generate and arrange high-quality content without specialized knowledge, and also provides optimal items that take into account the user's emotions.

[1410] The processing flow will be explained below.

[1411] Step 1:

[1412] The user enters a request

[1413] The user uses the device interface to input a request for the item they want to create. For example, to specify a "blue modern sofa," they input "sofa," "blue," and "modern." The device then converts this request into JSON-formatted data.

[1414] Step 2:

[1415] The device sends the requested data to the server

[1416] The terminal sends the converted request data to the server as an HTTP request. The server analyzes the received request data and converts it into a format that can be handled internally.

[1417] Step 3:

[1418] The server inputs data into the generated AI model and emotion engine

[1419] The server passes the analyzed request data to the generative AI model and emotion engine as input data. The generative AI model generates a 3D model and texture of the requested item based on the input data. Meanwhile, the emotion engine collects and analyzes data such as the user's facial expressions, voice, and text input to analyze the user's emotions.

[1420] Step 4:

[1421] Generative AI model and emotion engine generate items and emotion data

[1422] The generative AI model automatically generates a 3D model and texture of a blue modern sofa based on the specifications of "sofa," "blue," and "modern." Meanwhile, the emotion engine analyzes the user's input data and real-time emotion data to suggest the optimal items to satisfy the user. The generated item data and emotion data are returned to the server.

[1423] Step 5:

[1424] The server returns the generated results to the terminal

[1425] The server organizes the generated item data and the analysis results of the emotion engine and sends them back to the device, which then displays the generated results and suggestions to the user.

[1426] Step 6:

[1427] The user can review the generated results and suggestions and enter correction requests if necessary.

[1428] The user checks the generated items and the emotion engine's suggestions on the device. For example, if the color is different from what they expected, the user can input a correction request into the device, such as "make it a lighter blue." The device then converts this into JSON format and sends it to the server.

[1429] Step 7:

[1430] The server generates a correction request again and inputs it into the AI ​​model and emotion engine.

[1431] The server receives the user's modification request and again inputs the data into the generative AI model and emotion engine. The generative AI model generates a modified 3D model and texture based on the new input data, and the emotion engine generates new emotion data and suggestions. The generated data is returned to the server and sent back to the device.

[1432] Step 8:

[1433] The user confirms the item and confirms it

[1434] The user reviews the results again and repeats steps 6 and 7 as necessary until satisfied, finally finalizing the satisfactory items.

[1435] Step 9:

[1436] The server registers the item data in the metaverse space.

[1437] The server registers the confirmed item data in the metaverse space database. The item is reflected in the user's metaverse space.

[1438] Step 10:

[1439] Users place items in the metaverse space

[1440] The user logs in to the metaverse space using a terminal and places the generated items wherever they like. The user adjusts the position and orientation of the items to complete the final layout.

[1441] Example 2

[1442] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1443] In conventional systems, it was difficult for users to generate specific items while taking their emotions into consideration and providing optimal items. Furthermore, reflecting requests for revisions to generated items required manual work and multiple trial and error processes, which was inefficient. There was also a need for a system that could centrally manage these processes and be intuitive for users to operate.

[1444] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1445] In this invention, the server includes means for processing requests received from users and generating input data for generation, means for automatically generating items based on the input data using a generative model, means for providing the generated item data to the user and generating the item again in response to a request for confirmation and correction, means for recognizing the user's emotions and adjusting the parameters of the generative model in accordance with the emotions, and means for registering and arranging the finalized item data in a virtual space. This allows users to efficiently generate and modify optimal items that reflect their emotions and arrange them in a virtual space.

[1446] "User" refers to a person who utilizes a terminal to input requests into the system and to review and modify generated items.

[1447] "Server" refers to a computer system that receives request data from a user and generates and modifies items using generative models and emotion recognition engines.

[1448] A "terminal" refers to a device operated by a user to input requests and communicate with a server. Specifically, this applies to smartphones and computers.

[1449] "Request data" refers to data obtained by converting detailed information about an item that a user inputs through a terminal into a different format.

[1450] A "generative model" refers to an artificial intelligence model that automatically generates 3D models and textures based on required data.

[1451] An "emotion recognition engine" refers to a software system that analyzes a user's facial expressions and voice to identify the user's emotions.

[1452] "Virtual space" refers to a three-dimensional computer-generated environment constructed using digital technology that users can manipulate and experience as if it were a physical space through an interface.

[1453] "Input data" refers to data used by the generative model based on user request data.

[1454] The system of the present invention automatically generates items in a virtual space based on user requests, recognizes the user's emotions, and suggests or modifies items according to the emotions. This system includes a series of processes: request input, request data processing, use of a generation AI model and an emotion recognition engine, provision of the generated results and reflection of modification requests, and final registration and placement of the item data in the virtual space.

[1455] First, a user uses a terminal to input a request for creating an item. For example, if a user wants to create a "blue modern sofa," the user inputs detailed instructions such as "sofa," "blue," and "modern" into the interface. The terminal converts these requests into an appropriate format (e.g., JSON format) and sends them to the server.

[1456] The server receives and analyzes the request data sent by the user. The analyzed data is passed to the generative AI model and emotion recognition engine. The generative AI model generates a 3D model and texture of the item based on this input data. Meanwhile, the emotion recognition engine recognizes the user's emotion and adjusts the parameters of the generative AI model. For example, if the user is smiling, the emotion recognition engine instructs the generated item to use bright colors. The generated item data is temporarily stored on the server.

[1457] The generated item data is sent from the server to the device for user confirmation. The user can review the generated results on the device and input correction requests as needed. For example, if the color of the sofa is darker than expected, the user can input a correction request such as "make it a lighter blue." The emotion recognition engine analyzes the user's emotions and the new request and suggests the optimal correction. The server then uses the generative AI model again based on the new instructions to generate the corrected item.

[1458] Finally, if the user is satisfied with the generated item, the server registers the item data in the virtual space. The registered item can then be placed and used by the user in the virtual space. The user can then adjust the position and orientation of the item in their virtual space to complete the room decoration.

[1459] Specific examples

[1460] A specific example prompt for a user to generate a "blue modern sofa" would be:

[1461] "Generate a blue modern sofa."

[1462] "Please suggest a blue sofa that reflects the emotion you read from the user's facial expression."

[1463] This system allows users to easily create and arrange high-quality items without specialized knowledge, and also provides optimal items that take into account the user's emotions.

[1464] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1465] Step 1: User enters item creation request

[1466] The user inputs details of the item they want to create (e.g., "sofa," "blue," "modern") into the device interface. The device receives this as input data and converts it into JSON format using a format conversion tool. The input is the user's instructions, and the output is JSON format data.

[1467] Step 2: The terminal converts the requested data into a different format and sends it

[1468] The terminal converts the item creation request entered by the user into JSON format and then sends this data to the server. Here, the input is the detailed request data from the user, and the output is the JSON data sent to the server.

[1469] Step 3: Parsing and processing the request data by the server

[1470] The server receives the JSON data sent from the device and analyzes it. The analyzed content includes the item type, color, style, etc. The analyzed data is passed as input to the generative AI model and emotion recognition engine. The input is JSON data, and the output is the analyzed item information.

[1471] Step 4: Generate items using a generative AI model

[1472] The server passes the analyzed item information to the generative AI model, which generates a 3D model and texture according to the specified conditions (e.g., "blue modern sofa"). The generated data is temporarily stored on the server. The input is the analyzed item information, and the output is the generated 3D model and texture.

[1473] Step 5: Recognize and adjust the user's emotions using the emotion recognition engine

[1474] The server uses an emotion recognition engine to recognize the user's emotions. This involves analyzing the user's facial expressions and voice using the device's camera and microphone. The input is the user's facial expression and voice data, and the output is the recognized emotional information. This emotional information is used to adjust the parameters of the generative AI model and reflect it in the output.

[1475] Step 6: Server saves and sends the generated results

[1476] The server temporarily stores the generated and adjusted item data and then transmits it to the user's device. The input is the adjusted item data, and the output is the data sent to the device.

[1477] Step 7: User checks the generated results and requests corrections

[1478] The user checks the generated item on the terminal interface and inputs correction requests as needed. For example, the user may request "make it a lighter blue." The input is the generated item data, and the output is the user's correction requests.

[1479] Step 8: Emotion Recognition Engine Suggests Corrections

[1480] The emotion recognition engine proposes optimal corrections based on the user's new request and the results of facial and voice analysis. The input is the user's new request and emotional data, and the output is the optimal correction proposal.

[1481] Step 9: Server creates and resubmits the corrected item

[1482] The server uses the generative AI model again based on the user's correction request to generate a corrected item. This corrected item is then resent to the user's device. The input is the correction instruction and emotion data, and the output is the corrected item data.

[1483] Step 10: Register and place items in the metaverse space

[1484] The server registers the items that the user is finally satisfied with in the virtual space. After registration is complete, the user can freely place items in the virtual space and customize the environment. The input is the finalized item data, and the output is the items that have been registered in the virtual space and can now be placed.

[1485] (Application example 2)

[1486] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1487] Previously, when creating items in a virtual store or metaverse space, users had to manually input and edit data using a limited interface, which was extremely time-consuming and laborious. Furthermore, if the created item did not match the user's emotions or preferences, the user would have to re-edit it, resulting in a poor user experience. The present invention aims to solve these problems and provide a system that allows users to create and edit items more comfortably and efficiently.

[1488] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing the user's emotions in real time and adjusting the parameters of the generation AI model based on the emotion data, means for providing the generated item data to the user and generating items again in response to confirmation and correction requests, and means for registering and arranging the finalized item data in the virtual space. This allows the user to easily generate and modify optimal items based on their emotions and preferences, and efficiently arrange them.

[1489] "Means for recognizing a user's emotions in real time and adjusting the parameters of a generative AI model based on the emotion data" refers to a system that has the ability to recognize emotions in real time not only from data entered by the user but also from the user's facial expressions, voice, etc., and dynamically adjust the operating parameters of a generative AI model according to those emotions.

[1490] A "generative AI model" is an artificial intelligence model that automatically generates specific items based on input data, including processes that generate 3D models and textures.

[1491] The "emotion engine" analyzes the user's emotions and suggests or modifies items based on that emotion data, thereby providing the optimal items that the user desires.

[1492] "Virtual space" refers to a virtual area where users can engage in various activities within a digital space, such as a metaverse environment or a virtual store provided via the Internet.

[1493] The "means for generating an item again in response to a confirmation and correction request" is a system function that, when a user confirms a generated item and requests correction as necessary, generates the item again based on that request.

[1494] "Means for registering and arranging finalized item data in a virtual space" is a function that allows a user to register items that have been confirmed to their satisfaction in a virtual space and arrange them freely within that virtual space.

[1495] The "format conversion means for converting a user's request into input data" is a means having the function of appropriately formatting a request input by a user and converting it into a form that is easily understood by the system.

[1496] The system according to the present invention automatically generates items in a virtual space based on requests from a user, and further recognizes the user's emotions and suggests or modifies items according to the emotions.

[1497] System Program

[1498] Basic configuration

[1499] User terminal: A device used by users to input requests and check generated items. This can be a smart glass or a smartphone.

[1500] Server: The central device that processes request data and generates items using generative AI models and emotion engines.

[1501] Generative AI model: An artificial intelligence model that generates items based on user requests.

[1502] Emotion engine: An engine that analyzes user emotion data and adjusts the parameters of the generative AI model based on that data.

[1503] Explanation of program processing

[1504] The server first receives a request entered by the user via their device. This request is converted into a data format such as JSON. The server then inputs the converted data into a generative AI model to generate a 3D model and texture of the item based on the request. The emotion engine also recognizes emotional data from the user's facial expressions, voice, and text input, and adjusts the parameters of the generative AI model.

[1505] The generated item is immediately sent to the user's terminal, where the user can confirm it. If the user requires any modifications, the request is sent again to the server. This allows the optimal item to be generated according to the user's feelings. Modifications continue until the user is satisfied, and the finalized item is registered in the virtual space.

[1506] Specific examples

[1507] Suppose a user accesses a virtual store using smart glasses and enters a request by voice, such as "a blue modern sofa." The keywords "sofa," "blue," and "modern" are transmitted to the server through the interface of the smart glasses.

[1508] The server receives this and generates a 3D model and texture of a "blue modern sofa." At the same time, the emotion engine detects a "relaxed atmosphere" from the user's smile and adjusts the color of the sofa slightly brighter accordingly. The generated result is presented to the user, and if the user requests a correction such as "a darker blue," the item is generated again using the same process.

[1509] Example prompt sentence:

[1510] "Create a modern blue sofa that has a relaxing atmosphere."

[1511] This system allows users to easily create items that best suit their emotions and preferences, and efficiently place them in the virtual space.

[1512] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1513] Step 1:

[1514] A user uses a device (such as smart glasses or a smartphone) to input requests such as "sofa," "blue," and "modern" into the user interface. This input request data is converted into a data format such as JSON and sent to the server.

[1515] Input: User request (e.g. "sofa", "blue", "modern")

[1516] Output: Formatted request data (e.g., JSON format)

[1517] Specific operation:

[1518] When a user inputs something using voice or gestures, the device analyzes it and converts it into text data, which it then converts into JSON format and sends to the server via the network.

[1519] Step 2:

[1520] The server receives and analyzes the request data sent from the device. The analyzed data is input into the generative AI model to generate a 3D model and texture of a "blue modern sofa." The generated temporal data is also adjusted by the emotion engine based on the user's emotional data.

[1521] Input: Formatted request data

[1522] Output: Generated item data (3D models and textures)

[1523] Specific operation:

[1524] The server analyzes the received JSON data and passes the keywords "sofa," "blue," and "modern" to the generative AI model. The generative AI model creates a 3D model and texture based on the data. At the same time, the emotion engine analyzes the user's emotion data and adjusts the parameters of the generative AI model to generate the optimal output.

[1525] Step 3:

[1526] The generated item data is temporarily stored and sent to the user's terminal. The user can check the generated item on the terminal and, if not satisfied, input a correction request. This correction request is sent back to the server.

[1527] Input: Generated item data

[1528] Output: Correction requests from the user

[1529] Specific operation:

[1530] The item data is sent from the server to the device, where the user confirms it. If the user wishes to make any changes, they can enter their request by voice or text, such as "make it a lighter blue." The device then converts this request back into JSON format and sends it to the server.

[1531] Step 4:

[1532] The server receives the modification request from the user again and generates the item again using the generative AI model and emotion engine, thereby providing an item optimized for the user's emotions.

[1533] Input: User request for correction (e.g., "Make it a lighter blue")

[1534] Output: Regenerated item data

[1535] Specific operation:

[1536] The server receives the correction request, inputs it back into the generative AI model as data, and uses the emotion engine to readjust it based on the user's emotions. The generative AI model then creates new 3D models and textures and sends the data to the user's device.

[1537] Step 5:

[1538] Finally, once an item that satisfies the user is created, the server registers and places the item data in the virtual space. The user can then adjust the position and orientation of the item within the virtual space to complete the decoration of the space.

[1539] Input: Finalized item data

[1540] Output: Items registered in the virtual space

[1541] Specific operation:

[1542] The server receives the item data that the user is ultimately satisfied with and registers it in the specified location within the virtual space. The user can then freely adjust the position and orientation of the items within the virtual space via their device to complete the decoration of the space.

[1543] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1544] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1545] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1546] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1547] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1548] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1549] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1550] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1551] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1552] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1553] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1554] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1555] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1556] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1557] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1558] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1559] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1560] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1561] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1562] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1563] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1564] The following is further disclosed regarding the above embodiment.

[1565] (Claim 1)

[1566] means for processing requests received from users and generating input data for item creation;

[1567] means for automatically generating items based on the input data using a generative AI model;

[1568] a means for providing the generated item data to a user and generating the item again in response to a request for confirmation and correction;

[1569] A system including a means for registering and placing finalized item data in a metaverse space.

[1570] (Claim 2)

[1571] 10. The system of claim 1, further comprising a format conversion means for converting a user request into input data.

[1572] (Claim 3)

[1573] 10. The system of claim 1, wherein the generative AI model includes a process for generating 3D models and textures.

[1574] "Example 1"

[1575] (Claim 1)

[1576] means for processing requests received from users and generating input data for item creation;

[1577] A means for converting a user's request into a JSON format using a terminal and transmitting the JSON format to a server;

[1578] A means for the server to analyze the JSON data received and format it as input data for the generative AI model;

[1579] means for automatically generating items based on the input data using a generative AI model;

[1580] A means for temporarily storing the generated item data on a server;

[1581] a means for providing the generated item data to a user and generating the item again in response to confirmation and correction requests from the user;

[1582] A means for registering and arranging item data that a user is satisfied with in a metaverse space;

[1583] A system that includes a means for adjusting the position and orientation of registered items within a metaverse space.

[1584] (Claim 2)

[1585] 10. The system of claim 1, further comprising a format conversion means for converting a user request into input data.

[1586] (Claim 3)

[1587] 10. The system of claim 1, wherein the generative AI model includes a process for generating 3D models and textures.

[1588] "Application Example 1"

[1589] (Claim 1)

[1590] means for processing requests received from users and generating input data for item creation;

[1591] means for automatically generating items based on the input data using a generative AI model;

[1592] a means for providing the generated item data to a user and generating the item again in response to a request for confirmation and correction;

[1593] A means for registering and placing the finalized item data in the metaverse space;

[1594] A system including means for providing a visual interface for placing and customizing generated items in a virtual storefront.

[1595] (Claim 2)

[1596] 10. The system of claim 1, further comprising a format conversion means for converting a user request into input data.

[1597] (Claim 3)

[1598] 10. The system of claim 1, wherein the generative AI model includes a process for generating 3D models and textures.

[1599] "Example 2: Combining Emotion Engines"

[1600] (Claim 1)

[1601] means for processing requests received from a user and generating input data for generation;

[1602] means for automatically generating items based on the input data using a generative model;

[1603] a means for providing the generated item data to a user and generating the item again in response to a request for confirmation and correction;

[1604] a means for recognizing a user's emotion and adjusting the parameters of a generative model according to the emotion;

[1605] A system including a means for registering and placing finalized item data in a virtual space.

[1606] (Claim 2)

[1607] 10. The system of claim 1, further comprising a format conversion means for converting a user request into input data.

[1608] (Claim 3)

[1609] 10. The system of claim 1, wherein the generative model includes a process for generating a 3D model and texture.

[1610] "Application example 2 when combining emotion engines"

[1611] (Claim 1)

[1612] means for processing requests received from users and generating input data for item creation;

[1613] means for automatically generating items based on the input data using a generative AI model;

[1614] A means for recognizing a user's emotions in real time and adjusting the parameters of the generative AI model based on the emotion data;

[1615] a means for providing the generated item data to a user and generating the item again in response to a request for confirmation and correction;

[1616] A means for registering and arranging the finalized item data in the virtual space;

[1617] ...

[1618] A system including:

[1619] (Claim 2)

[1620] 10. The system of claim 1, further comprising a format conversion means for converting a user request into input data.

[1621] (Claim 3)

[1622] 10. The system of claim 1, wherein the generative AI model includes a process for generating 3D models and textures.

[1623] (Claim 4)

[1624] 10. The system of claim 1, further comprising an emotion engine that analyzes requests and emotion data from a user and makes optimal suggestions or modifications. [Explanation of symbols]

[1625] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for processing requests received from users and generating input data for item creation; means for automatically generating items based on the input data using a generative AI model; a means for providing the generated item data to a user and generating the item again in response to a request for confirmation and correction; A system including a means for registering and placing finalized item data in a metaverse space.

2. 2. The system of claim 1, further comprising a format conversion means for converting a user request into input data.

3. 10. The system of claim 1, wherein the generative AI model includes a process for generating 3D models and textures.

Images