System

Abstract

A system is provided.SOLUTION: This system is provided with a means for designating an object desired by a user by a prompt or an image, a means for generating a Python script by a generation AI based on the designated prompt or image, a means for preparing the object by 3D modeling software by using the generated Python script, and a means for arranging the prepared object in a virtual space.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Building a metaverse space requires high technical skills and specialized knowledge, making it difficult for general users to easily perform 3D modeling. Furthermore, even if a company were to build a metaverse space, it would be difficult to operate in a practical manner due to technical and budgetary constraints and the heavy workload. Furthermore, in a remote work environment, lack of communication between employees and feelings of loneliness are issues. [Means for solving the problem]

[0005] This invention provides a means for users to easily specify a desired object using a prompt or image. It also provides a means for a generation AI to automatically generate a Python script based on the specified prompt or image, and for the script to be used to create an object in 3D modeling software. It also provides a means for users to easily place the generated object in a virtual space, allowing them to participate in the virtual space as an avatar and promoting communication through dialogue with non-player characters (NPCs) and interactions between NPCs.

[0006] A "prompt" is a text input that allows the user to instruct the generated AI in natural language about the desired object's characteristics and shape.

[0007] An "image" is a means by which the user visually indicates the desired object to the generating AI, and is uploaded in a format such as JPEG or PNG.

[0008] "Generative AI" is artificial intelligence that automatically generates Python scripts based on user prompts and images.

[0009] A "Python script" is program code generated by generative AI to run in 3D modeling software.

[0010] "3D modeling software" refers to software for creating and editing 3D objects by executing generated Python scripts.

[0011] An "object" is a 3D model placed in a virtual space and generated based on user prompts and images.

[0012] A "virtual space" is a digital environment in which users can participate as avatars, and is an area in which 3D objects are placed.

[0013] An "avatar" is a digital character that represents a user in a virtual space.

[0014] An "NPC (Non-Player-Character)" is a non-operable character that is placed in a virtual space and interacts with the user and other NPCs. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The system of the present invention allows the user to specify the desired object using prompts or images, automatically generates a Python script using generation AI, creates the object using 3D modeling software, and places the object in virtual space.

[0037] Object Creation

[0038] First, a user accesses the Metaverse administration screen and selects the object creation option. The user enters a prompt for the desired object's characteristics and shape, or uploads an existing image. For example, a user might enter the prompt "wooden bench."

[0039] The server then receives the user's input and provides prompts and image data to the generation AI, which then creates a Python script to generate the object. This Python script, when executed in 3D modeling software, contains the specific steps and parameters required to generate the specified object.

[0040] The Python script generated by the generative AI is sent from the server to the terminal, and the terminal generates an object using 3D modeling software based on this script. Common 3D modeling software such as Blender can be used here.

[0041] Object placement in virtual space

[0042] The device also performs the procedure to place the generated object in the virtual space as a 3D model. This allows users to intuitively place the objects they have created in the virtual space and share them with other users. For example, a generated "wooden bench" can be placed in the lobby of a virtual office.

[0043] Realization of in-house office

[0044] The server then uses the AI ​​model to generate non-player characters (NPCs). These NPCs have various roles, such as acting as advisors in meetings or as idea-boosting assistants. The generated NPC data is sent from the server to the device, which can then use it to place the NPCs in the virtual space.

[0045] User participation and interaction

[0046] Finally, users participate in the virtual space as avatars. Users can select their own avatars, log in, and interact with generated objects and NPCs. In particular, NPCs not only interact with users, but also with each other, providing new ideas and recreating the lively atmosphere of an office.

[0047] Specific examples

[0048] For example, a user selects "Object Creation" on the Metaverse management screen and enters the prompt "wooden bench." The server sends this prompt to the generation AI, which generates a Python script. The server then sends the script to the device, which then runs the script in Blender to generate a 3D model of the "wooden bench." The generated model is placed in the office lobby in the virtual space.

[0049] Meanwhile, the server uses an AI model to generate an NPC that acts as an "idea-boosting assistant" and sends that data to the device. The device places this NPC in the virtual space, and the user logs in as an avatar and interacts with the NPC, allowing the idea-boosting process to take place in the virtual space.

[0050] As described above, the system of the present invention enables users to easily generate and place 3D objects and recreate the experience of a real-world office in a virtual space.

[0051] The processing flow will be explained below.

[0052] Object creation process flow

[0053] Step 1:

[0054] A user accesses the Metaverse administration screen and selects "Create Object," where they are prompted to enter the desired object's characteristics and shape, or upload an existing image.

[0055] Step 2:

[0056] The server receives input data (prompts or images) from the user and sends it to the generation AI, which then automatically creates a Python script to generate an object based on the specified prompt or image.

[0057] Step 3:

[0058] The server sends the generated Python script to the user's device, which receives it and saves it to its local disk.

[0059] Step 4:

[0060] The terminal runs the saved Python script in 3D modeling software (e.g., Blender), which generates a 3D object based on the specified properties and shape.

[0061] Object placement in virtual space

[0062] Step 5:

[0063] The device performs a procedure for placing the generated 3D object in a virtual space, specifically, exporting or placing the object generated using the 3D modeling software at a specified location in the virtual space.

[0064] Realization of in-house office

[0065] Step 1:

[0066] The server uses the AI ​​model to generate an NPC with the role specified by the user (for example, an idea-boosting assistant). This NPC has the data necessary to interact with the user and communicate with other NPCs.

[0067] Step 2:

[0068] The server sends the generated NPC data to the user's device, which receives the data and places it in the virtual space.

[0069] Step 3:

[0070] Once the device has completed placing the NPC, the NPC becomes active in the virtual space, allowing for conversation and interaction with the user and other NPCs.

[0071] User participation and interaction

[0072] Step 1:

[0073] A user logs into a virtual space as an avatar. The user selects their avatar and completes the login process.

[0074] Step 2:

[0075] The user can start interacting with the generated objects and NPCs. For example, they can provide ideas to NPCs placed as idea-boosting assistants and receive advice or new input in response.

[0076] As described above, the system of the present invention realizes a series of processes that allow users to easily generate and place 3D objects and recreate a real-world office experience in a virtual space.

[0077] Example 1

[0078] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0079] Conventional systems for generating 3D objects and placing them in virtual spaces are difficult to use unless the user has a high level of expertise. Furthermore, the interaction functions with generated objects and NPCs (Non-Player-Characters) are insufficient, resulting in poor user experience and reduced usability. Furthermore, security and processing speed issues frequently arise when exchanging generated data.

[0080] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0081] In this invention, the server includes an input means for a user to input a prompt or an image, a means for a generative AI model to generate a Python script based on the input prompt or image, and a means for creating an object with 3D modeling software using the generated Python script, which enables even users without specialized knowledge to easily generate 3D objects and intuitively place them in a virtual space.

[0082] In this invention, the server further includes a means for allowing users to participate in the virtual space as avatars, a means for generating and placing NPCs (Non-Player-Characters) in the virtual space and for the NPCs to interact with the user, and a means for the user to refer to information generated by the interactions between NPCs, thereby enabling richer interactions in the virtual space and improving the user experience.

[0083] This invention also includes a means for using a combination of an image generation algorithm and a 3D modeling algorithm to generate an object using a generative AI model, a means for transmitting the generated 3D model to a user's terminal via a server, and a means for placing the transmitted 3D model in a virtual space on the user's terminal, thereby enabling the generated data to be exchanged efficiently and safely and quickly placed in the virtual space.

[0084] "prompt"

[0085] is an input format in which a user specifies the desired properties and shape of an object in text format.

[0086] "image"

[0087] is visual data that a user uploads to visually specify an existing object.

[0088] "Generative AI model"

[0089] is a system that includes Python scripts and artificial intelligence algorithms for generating 3D objects based on user input data.

[0090] "Python script"

[0091] is program code that is automatically generated by a generative AI model and contains instructions and parameters for creating an object in 3D modeling software.

[0092] "3D modeling software"

[0093] is a software application that runs Python scripts to create and edit 3D objects. Examples include Blender.

[0094] "Object"

[0095] is a digital model with three-dimensional shape and structure that is generated to be placed in a virtual space.

[0096] "Virtual Space"

[0097] is a computer-generated environment built for users to interact with digital objects and NPCs.

[0098] "Users"

[0099] A player is a person who uses the system to input prompts or images to interact with generated objects and NPCs.

[0100] "Input means"

[0101] is the interface through which the user provides prompts and images to the system.

[0102] "Generation means"

[0103] is a component that performs processing to generate an object based on the input prompt and image.

[0104] "Method of Creation"

[0105] is a method for running the generated Python script to create the object in 3D modeling software.

[0106] "Arrangement means"

[0107] is a tool or process for properly positioning generated objects in a virtual space.

[0108] "Avatar"

[0109] is a digital character that users use to represent themselves in a virtual space.

[0110] "NPC (Non-Player-Character)"

[0111] is a digital character generated by the system that interacts with the user in a virtual space.

[0112] "Image generation algorithm"

[0113] is an algorithm that manages the process of generating objects based on user-provided image data.

[0114] "3D modeling algorithm"

[0115] is an algorithm for calculating and generating 3D shapes used by generative AI models.

[0116] "Transmission means"

[0117] is a protocol and function for transferring generated 3D models and data from a server to a user's device.

[0118] The system of the present invention allows the user to specify the desired object using prompts or images, automatically generates a Python script using a generative AI model, creates the object using 3D modeling software, and places the object in a virtual space.

[0119] Hardware and Software Configuration

[0120] Server: A high-performance server machine is used to run the generative AI model and process the data.

[0121] Terminal: Use a personal computer to accept user input, run generated Python scripts, and place 3D objects in the virtual space.

[0122] Generative AI model: An artificial intelligence model for generating Python scripts based on user prompts and images. For example, we use a natural language processing model (e.g., ChatGPT).

[0123] 3D modeling software: Software that runs Python scripts to create and edit 3D objects. Blender will be used as an example.

[0124] Object creation procedure

[0125] 1. A user accesses the Metaverse Admin screen and selects the object creation option. Here, the user enters the desired object characteristics and shape as a prompt, or uploads an existing image. In this example, the user enters the prompt "wooden bench."

[0126] 2. The server receives the user's input data (prompts and images) and provides the prompts and image data to the generative AI model. The generative AI model then creates a Python script to generate the object. This Python script contains the specific steps and parameters to be executed in the 3D modeling software.

[0127] 3. The generated Python script is sent from the server to the terminal, which then launches 3D modeling software such as Blender and executes the Python script to generate the specified object.

[0128] Object placement in virtual space

[0129] The device then performs the procedure to place the generated 3D object in the virtual space. Specifically, the user can intuitively drag and drop the object within the virtual space to place it. For example, placing a generated "wooden bench" in the lobby of a virtual office corresponds to this operation.

[0130] Realization of in-house office

[0131] The server uses AI models to generate non-player characters (NPCs). These NPCs have various roles, such as acting as advisors in meetings or as idea-boosting assistants. The generated NPC data is then sent from the server to the device, which can then use it to place the NPCs in the virtual space.

[0132] User participation and interaction

[0133] Users participate in the virtual space as avatars. They can select their own avatar, log in, and interact with generated objects and NPCs. In particular, NPCs not only interact with users, but also with each other, providing new ideas and recreating the lively atmosphere of an office.

[0134] Specific examples

[0135] As a specific example, a case will be described in which a user selects "Create Object" on the metaverse management screen and inputs the prompt "Wooden Bench."

[0136] 1. The server sends this prompt to the generation AI, which generates a Python script.

[0137] 2. The generated script is sent from the server to the device, and the device executes the script in Blender to generate a 3D model of a "wooden bench."

[0138] 3. The user drags and drops the model into the virtual office lobby.

[0139] As another specific example, we will also explain a case where the server uses an AI model to generate an NPC with the role of an "idea bouncing assistant."

[0140] 1. The generated NPC data is sent from the server to the device.

[0141] 2. The device places this NPC in the virtual space, and the user logs in as an avatar and interacts with the NPC, bouncing ideas off each other in the virtual space.

[0142] As described above, the system of the present invention allows even users without specialized knowledge to easily generate 3D objects and intuitively place them in a virtual space.

[0143] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0144] Step 1:

[0145] The user accesses the metaverse management screen and selects the option for creating the object they desire.

[0146] Input: User access and object creation choices

[0147] Output: Object creation option selection

[0148] Step 2:

[0149] The user enters the desired object characteristics or shape as a prompt, or uploads an existing image. For example, enter the prompt "wooden bench."

[0150] Input: prompt or image data

[0151] Output: Prompt or image data is reflected in the Metaverse management screen

[0152] Step 3:

[0153] The server receives the user's input data (prompts and images).

[0154] Input: prompt or image data

[0155] Output: State that receives prompt or image data

[0156] Step 4:

[0157] The server provides prompts and image data to the generated AI model.

[0158] Input: prompt or image data

[0159] Output: Providing data to a generative AI model

[0160] Step 5:

[0161] Create a Python script that will allow the generative AI model to generate objects based on the provided prompts and image data, with specific steps and parameters to be executed in the 3D modeling software.

[0162] Input: prompt or image data

[0163] Output: Python script

[0164] Step 6:

[0165] The server sends the generated Python script to the device.

[0166] Input: Python script

[0167] Output: Script sending status to the terminal

[0168] Step 7:

[0169] Based on the Python script received by the device, 3D modeling software such as Blender is launched.

[0170] Input: Python script

[0171] Output: 3D modeling software running

[0172] Step 8:

[0173] The terminal will run the Python script in Blender and create the specified object.

[0174] Input: Python script

[0175] Output: Generated 3D object

[0176] Step 9:

[0177] The device provides an interface for placing the generated 3D objects in the virtual space. Users can intuitively drag and drop objects to place them in the virtual space. For example, placing a wooden bench in the lobby of a virtual office.

[0178] Input: Generated 3D object

[0179] Output: Objects placed in virtual space

[0180] Step 10:

[0181] The server generates NPCs (Non-Player-Characters) using an AI model and sends the data to the device.

[0182] Input: None (internal data)

[0183] Output: Send NPC data to the terminal

[0184] Step 11:

[0185] The device places NPCs generated by an AI model into the virtual space.

[0186] Input: NPC data

[0187] Output: NPC placed in virtual space

[0188] Step 12:

[0189] Users participate in the virtual space as avatars. Users select their own avatar and log in.

[0190] Input: Avatar selection and login information

[0191] Output: Avatar participation status in virtual space

[0192] Step 13:

[0193] Users interact with generated objects and NPCs through conversations. By observing the conversations and information sharing between NPCs, new ideas can be gained.

[0194] Input: User actions and interactions

[0195] Output: Dialogue results and new ideas

[0196] (Application example 1)

[0197] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0198] In modern virtual spaces and virtual stores, it is difficult for users to intuitively customize desired products and objects and then view and operate the generated objects. In particular, there is a demand for a system that allows users to easily generate specific objects without specialized knowledge and view and operate them in a virtual space. Another challenge is the placement of automatic characters to enhance user interaction and provide a more realistic experience, as well as the utilization of information generated by these characters.

[0199] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0200] In this invention, the server includes a means for a user to specify a desired object using a prompt or image, a means for a generation AI to generate a program based on the specified prompt or image, and a means for creating an object using 3D modeling software using the generated program. This allows users to easily generate desired objects without any special knowledge, place them in a virtual space, and view and manipulate them. Furthermore, to enhance the user's experience, automated characters can be placed and information generated through interactions between the user and the characters can be provided.

[0201] A "prompt" is a text input that allows a user to specify the desired properties or shape of an object.

[0202] An "image" is visual data uploaded by a user to specify the properties and shape of a desired object.

[0203] A "program" is a script automatically generated by generative AI to create an object in 3D modeling software.

[0204] "3D modeling software" refers to software tools for programmatically generating 3D objects.

[0205] A "virtual space" is a computer-generated space that users can interact with.

[0206] A "user" is an entity that uses the system to specify and create a desired object.

[0207] A "terminal" is a digital device used by a user, such as a computer or smartphone.

[0208] "Automatic characters" refer to non-player characters (NPCs) that can interact with users in a virtual space and with each other.

[0209] "Generative AI" is an artificial intelligence technology that automatically creates programs to generate objects based on input prompts and images.

[0210] An "algorithm" is a step-by-step procedure or formula for solving a particular problem.

[0211] The "information processing device" is a network-compatible device for transmitting the generated three-dimensional model to the user's terminal.

[0212] A "model" is generated three-dimensional object data, and refers to a specific design that is placed in a virtual space.

[0213] An "object" refers to any object or item designated by the user, and is a target object generated in a virtual space.

[0214] The embodiment of the present invention is based on a system comprising the following steps.

[0215] First, the user accesses the virtual environment management screen using a smartphone or personal computer. This management screen provides multiple options for creating the object the user desires. The user specifies the desired object by entering a prompt or uploading an image. Examples of prompts include a "wooden bench" or a "stylish glass table."

[0216] The user's input data is then sent to a server, which uses generative AI to automatically generate a program to generate a concrete object from the prompt or image. This program is written as a Python script and can be executed in 3D modeling software (e.g., Blender).

[0217] The generated Python script is sent to the user's terminal via the network, where it is executed by the 3D modeling software to generate a specific object.

[0218] The generated 3D model is then placed directly in a virtual space on the device, allowing the user to visually confirm the object of their choice within the virtual space. This virtual space can be a virtual store, and the user can place the generated object in real life for use in sales promotions, interior design simulations, and more.

[0219] Furthermore, the server can generate automated characters and place them in the virtual space. This allows users to interact with the automated characters, resulting in a more realistic experience. The automated characters can also interact with each other, and users can view the content of their conversations and the information generated.

[0220] Hardware and software used

[0221] Server: A computer server for running generative AI

[0222] Device: A smartphone or personal computer operated by a user

[0223] 3D modeling software: Blender, etc.

[0224] Generative AI: Artificial intelligence technology that generates programs based on prompts and images

[0225] Specific examples

[0226] 1. The user selects "Object Creation" on the virtual space management screen and enters the prompt "Wooden bench."

[0227] 2. The server sends this prompt to the generation AI, which generates a Python script.

[0228] 3. The generated script is sent from the server to the device, and the device executes the script in Blender to generate a three-dimensional model of the "wooden bench."

[0229] 4. The generated model is placed in the virtual store showroom.

[0230] In this way, the present invention enables users to easily create and place desired objects and freely manipulate them in virtual space without any special knowledge.

[0231] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0232] Step 1:

[0233] A user accesses the virtual space management screen and selects the "Object Creation" option. The user is prompted to enter the characteristics and shape of an object, such as a "wooden bench," or upload an existing image.

[0234] Input: User input (prompt or image)

[0235] Output: Input prompt and image data

[0236] Step 2:

[0237] The server receives user input and sends the data to the Generative AI, which generates a Python script to generate the object based on the prompt and image.

[0238] Input: User input (prompt or image)

[0239] Data processing: Calculation process to generate Python scripts based on prompt statements

[0240] Output: Generated Python script

[0241] Step 3:

[0242] The generated Python script is sent from the server to the user's terminal.

[0243] Input: Generated Python script

[0244] Output: Python script sent to user's terminal

[0245] Step 4:

[0246] The terminal executes the received Python script in 3D modeling software (e.g., Blender) to generate an object.

[0247] Input: The received Python script

[0248] Data processing: 3D model generation processing using 3D modeling software

[0249] Output: Generated 3D model

[0250] Step 5:

[0251] The generated three-dimensional model is placed in a virtual space on the terminal, and the user can visually check and operate it.

[0252] Input: Generated 3D model

[0253] Output: 3D model placed in the virtual space on the device

[0254] Step 6:

[0255] The server uses a generation AI to generate automated characters and place them in the virtual space, allowing users to interact with them.

[0256] Input: Automatic character generation request

[0257] Data processing: AI calculation processing for automatic character generation

[0258] Output: Generated auto character

[0259] Step 7:

[0260] The automated characters converse with the user and with each other, and the content of the conversation can be viewed by the user.

[0261] Input: User interaction with automated character

[0262] Output: Interactions and generated information

[0263] Step 8:

[0264] The generated information is sent to the user's terminal, where the user can check the details.

[0265] Input: Generated information

[0266] Output: Display information on the user's terminal

[0267] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0268] The system of the present invention allows users to specify the desired object using prompts or images, automatically generates a Python script using a generation AI, creates the object using 3D modeling software, and places the object in a virtual space. Furthermore, it combines an emotion engine that recognizes the user's emotions to optimize interactions within the virtual space.

[0269] Object Creation

[0270] First, a user accesses the Metaverse administration screen and selects the object creation option. The user enters a prompt for the desired object's characteristics and shape, or uploads an existing image. For example, a user might enter the prompt "wooden bench."

[0271] The server then receives the user's input and provides prompts and image data to the generation AI, which then creates a Python script to generate the object. This Python script, when executed in 3D modeling software, contains the specific steps and parameters required to generate the specified object.

[0272] The Python script generated by the generative AI is sent from the server to the terminal, and the terminal generates an object using 3D modeling software based on this script. Common 3D modeling software such as Blender can be used here.

[0273] Object placement in virtual space

[0274] The device also performs the procedure to place the generated object in the virtual space as a 3D model. This allows users to intuitively place the objects they have created in the virtual space and share them with other users. For example, a generated "wooden bench" can be placed in the lobby of a virtual office.

[0275] Realization of in-house office

[0276] The server then uses the AI ​​model to generate non-player characters (NPCs). These NPCs have various roles, such as acting as meeting advisors or idea-boosting assistants. The generated NPC data is sent from the server to the device, which can then use it to place the NPCs in the virtual space.

[0277] User participation and interaction

[0278] Users participate in the virtual space as avatars. They can select their own avatar, log in, and interact with generated objects and NPCs. In particular, NPCs not only interact with users, but also with each other, providing new ideas and recreating the lively atmosphere of an office.

[0279] Applying the Emotion Engine

[0280] The system of the present invention further incorporates an emotion engine that recognizes the user's emotions. The emotion engine analyzes the user's voice data and facial expression data in real time to identify the user's emotional state. For example, if the user looks happy, the emotion engine will recognize this as "joy."

[0281] Based on the analysis results, the server adjusts the NPC's responses and behavior. For example, if the user looks dissatisfied, the NPC will respond by offering words of encouragement or suggesting solutions. The emotion engine's analysis results are also reflected in the dialogue between NPCs, resulting in more plausible interactions.

[0282] Specific examples

[0283] For example, a user selects "Object Creation" on the Metaverse management screen and enters the prompt "wooden bench." The server sends this prompt to the generation AI, which generates a Python script. The server then sends the script to the device, which then runs the script in Blender to generate a 3D model of the "wooden bench." The generated model is placed in the office lobby in the virtual space.

[0284] Meanwhile, the server uses an AI model to generate an NPC that acts as an "idea-boosting assistant" and sends that data to the device. The device places this NPC in the virtual space, and the user logs in as an avatar and interacts with the NPC, allowing the idea-boosting process to take place in the virtual space.

[0285] Furthermore, if the user expresses dissatisfaction during the interaction, the emotion engine will analyze and recognize this information in real time, and the server will adjust the NPC's response accordingly, providing specific advice and solutions to the user's problems. In this way, the emotion engine optimizes the interaction and improves the user experience.

[0286] As described above, the system of the present invention allows users to easily generate and place 3D objects, realizing a series of processes that recreate a real-world office experience in a virtual space. Furthermore, by combining it with an emotion engine, it is possible to optimize interactions and increase user satisfaction.

[0287] The processing flow will be explained below.

[0288] Object creation process flow

[0289] Step 1:

[0290] A user accesses the Metaverse administration screen and selects "Create Object." They are prompted to enter the desired object's characteristics and shape, or they can upload an existing image.

[0291] Step 2:

[0292] The server receives input data (prompts or images) from the user and sends it to the generation AI, which then automatically creates a Python script to generate an object based on the specified prompt or image.

[0293] Step 3:

[0294] The server sends the generated Python script to the user's device, which receives it and saves it to its local disk.

[0295] Step 4:

[0296] The terminal runs the saved Python script in 3D modeling software (e.g., Blender), which generates a 3D object based on the specified properties and shape.

[0297] Processing flow for placing objects in virtual space

[0298] Step 5:

[0299] The device performs a procedure for placing the generated 3D object in a virtual space, specifically, exporting or placing the object generated using the 3D modeling software at a specified location in the virtual space.

[0300] Process flow for realizing in-house office

[0301] Step 1:

[0302] The server uses the AI ​​model to generate an NPC with the role specified by the user (for example, an idea-boosting assistant). This NPC has the data necessary to interact with the user and communicate with other NPCs.

[0303] Step 2:

[0304] The server sends the generated NPC data to the user's device, which receives the data and places it in the virtual space.

[0305] Step 3:

[0306] Once the device has completed placing the NPC, the NPC becomes active in the virtual space, allowing for conversation and interaction with the user and other NPCs.

[0307] User participation and interaction process flow

[0308] Step 1:

[0309] A user logs into a virtual space as an avatar. The user selects their avatar and completes the login process.

[0310] Step 2:

[0311] The user can start interacting with the generated objects and NPCs. For example, they can provide ideas to NPCs placed as idea-boosting assistants and receive advice or new input in response.

[0312] Process flow for applying the emotion engine

[0313] Step 1:

[0314] When a user interacts in a virtual space, the emotion engine collects the user's voice data and facial expression data in real time, analyzes this data, and identifies the user's emotional state.

[0315] Step 2:

[0316] The server receives the analysis results of the emotion engine and adjusts the NPC's response. For example, if the user has a dissatisfied expression, the NPC will respond by offering words of encouragement or suggesting a solution.

[0317] Step 3:

[0318] The server adjusts the dialogue between other NPCs based on the results of the emotion engine, enabling more natural communication within the virtual space.

[0319] Specific examples

[0320] A concrete example of object creation

[0321] For example, a user selects "Object Creation" on the Metaverse management screen and enters the prompt "wooden bench." The server sends this prompt to the generation AI, which generates a Python script. The server then sends the script to the device, which then runs the script in Blender to generate a 3D model of the "wooden bench." The generated model is placed in the office lobby in the virtual space.

[0322] Example of NPC generation and dialogue

[0323] The server uses an AI model to generate an NPC that acts as an "idea-boosting assistant" and sends that data to the device. The device places this NPC in the virtual space, and the user logs in as an avatar and interacts with the NPC, allowing the idea-boosting process to take place in the virtual space.

[0324] Specific examples of emotion engine application

[0325] If the user expresses dissatisfaction during the interaction, the emotion engine will analyze and recognize this information in real time, and the server will adjust the NPC's response accordingly, providing specific advice and solutions to the user's problems. In this way, the emotion engine optimizes the interaction and improves the user experience.

[0326] As described above, the system of the present invention allows users to easily generate and place 3D objects, realizing a series of processes that recreate a real-world office experience in a virtual space. Furthermore, by combining it with an emotion engine, the user experience can be further improved.

[0327] Example 2

[0328] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0329] Interaction in modern virtual environments requires a method that allows users to easily create and place objects. However, existing systems require specialized knowledge and numerous manual operations, and few are easy to use. Furthermore, there is a lack of means to accurately reflect the user's emotions and achieve more natural and effective interactions in virtual environments, limiting the quality of the user experience.

[0330] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0331] In this invention, the server includes: a prompt or image input means for a user to specify a desired object; a generation AI generating code based on the specified prompt or image; a 3D model creation software creating an object using the generated code; a virtual environment placing the created object; and a sentiment analysis engine recognizing the user's sentiment and adjusting the responses and behavior of a virtual character based on the recognition results. This allows users to easily create and place 3D objects without requiring specialized knowledge, and enables natural and effective dialogue within the virtual environment utilizing sentiment analysis.

[0332] A "prompt" is input data that describes the target or condition desired by the user in natural language.

[0333] An "image" is digital data that visually represents an object or condition specified by a user.

[0334] "Generative AI" is an artificial intelligence system that analyzes user input data and automatically generates the necessary code and models.

[0335] "Code" refers to the program instructions generated by the AI ​​based on prompts and images specified by the user.

[0336] "3D model creation software" is software that actually generates 3D objects based on generated code.

[0337] A "virtual environment" is a computer-generated simulated space in which a user can interact.

[0338] A "virtual character" is an NPC (non-player character) in a virtual environment, a character that is set up to converse and interact with the user.

[0339] An "emotion analysis engine" is a technology that analyzes a user's voice, facial expressions, etc. to identify their emotional state.

[0340] "User's computer" means an information processing device used by a user to receive data from the generated AI and use it within the virtual environment.

[0341] An "information processing device" is a device that transmits, receives, analyzes, and stores digital data, and examples of this include servers.

[0342] The present invention provides a system that allows a user to easily create desired objects and place them in a virtual environment. Specific embodiments of the system will be described below.

[0343] Hardware and software used

[0344] This system mainly uses the following hardware and software:

[0345] Server: Receives prompts and image data and sends them to the generation AI

[0346] Terminal: A computer operated by a user, with 3D modeling software such as Blender installed.

[0347] Generative AI model: Analyzes user input data and generates the necessary Python scripts

[0348] Virtual environment management software: Place and manage objects and virtual characters in a virtual environment

[0349] Emotion Analysis Engine: Analyzes user emotions and adjusts the virtual character's responses and behavior

[0350] Detailed System Description

[0351] 1. Object creation prompt input

[0352] A user opens a browser and logs into the Metaverse admin page, where they select the "Create Object" option and enter the desired object's characteristics and shape as a prompt, or upload an existing image (e.g., "Wooden Bench").

[0353] Example prompt sentence:

[0354] "Generate a wooden bench."

[0355] "Create a virtual character that acts as a meeting advisor."

[0356] 2. Generate prompts and send them to the AI

[0357] The server receives user input and provides prompts and image data to the generative AI, which then creates a Python script to generate the object. This script contains commands and parameters that can be executed in 3D modeling software such as Blender.

[0358] 3. Generate and run the Python script

[0359] The AI ​​analyzes the prompt and automatically generates a Python script, which is then sent to the user's device via a server. The device then uses 3D modeling software such as Blender to generate objects based on the script.

[0360] 4. Placing objects in the virtual environment

[0361] The device places the generated 3D model in the virtual environment, and the user can adjust the object's exact position using the mouse or keyboard. Information about the placed object is registered in the virtual environment management software and shared with other users.

[0362] 5. Virtual Character Generation and Emotional Response

[0363] The server uses AI models to generate virtual characters, which can act as meeting advisors, idea-boosting assistants, etc. These characters are sent to the device based on the data generated by the AI ​​and placed in the virtual environment.

[0364] In addition, an emotion analysis engine analyzes the user's voice and facial expressions in real time and adjusts the virtual character's responses and behavior accordingly. For example, if the user looks unhappy, the virtual character will offer words of encouragement or suggest solutions to the problem.

[0365] Specific examples

[0366] For example, consider the case of generating a "wooden bench." The user selects "Object Creation" on the metaverse management screen and enters "wooden bench" as a prompt. The server sends this prompt to the generation AI, which creates a Python script. The server then sends the script to the device, which then runs it in Blender. The resulting 3D model of the "wooden bench" is placed in the lobby of the virtual office.

[0367] In addition, a virtual character called a "meeting advisor" is generated, and when a user interacts with the virtual environment, an emotion analysis engine analyzes the user's emotional state in real time and adjusts the virtual character's responses accordingly, allowing users to experience more natural and effective interactions within the virtual environment.

[0368] The above is a concrete example of how to implement the invention. This system allows users to easily generate 3D objects without requiring specialized knowledge, and enables effective dialogue utilizing emotion analysis.

[0369] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0370] Step 1:

[0371] The user opens a browser and accesses the metaverse management screen. The login screen is displayed, the user enters their username and password, and clicks the login button. The entered username and password are sent to the server for authentication. If authentication is successful, the metaverse management screen is displayed.

[0372] Input: Username, Password

[0373] Output: Metaverse admin page

[0374] Step 2:

[0375] On the Metaverse Admin screen, select the "Create Object" option. The user enters the desired object's characteristics and shape as a prompt, or uploads an existing image. For example, enter the prompt "Wooden bench." The user's input is sent to the server.

[0376] Input: A prompt or image (e.g., "Wooden bench")

[0377] Output: Prompt data sent to the server

[0378] Step 3:

[0379] The server takes the user's input and sends it to a generative AI, which parses the prompt and generates a Python script containing commands and parameters that can be executed in 3D modeling software such as Blender.

[0380] Input: prompt data

[0381] Output: Python script

[0382] Step 4:

[0383] The generated Python script is sent from the server to the device, where it is executed in 3D modeling software such as Blender. The specified object is generated based on the script.

[0384] Input: Python script

[0385] Output: 3D object

[0386] Step 5:

[0387] The device places the generated 3D object in the virtual environment. The user can adjust the object's position using the mouse or keyboard. The information about the placed object is saved in the virtual environment management software and can be shared with other users.

[0388] Input: 3D object

[0389] Output: Objects placed in the virtual environment

[0390] Step 6:

[0391] The server uses an AI model to generate a virtual character, which can act as a meeting advisor or idea-boosting assistant, for example. The generated character data is sent from the server to the device, which then places it in the virtual environment.

[0392] Input: Virtual character role (e.g., conference advisor)

[0393] Output: Generated virtual character data

[0394] Step 7:

[0395] The device places a virtual character in the virtual environment. When a user logs into the virtual environment as an avatar, they can begin interacting with the virtual character. An emotion analysis engine analyzes the user's emotions in real time and adjusts the virtual character's responses and behavior based on the results.

[0396] Input: Virtual character data, user emotion data

[0397] Output: User interaction with virtual character

[0398] Through the above process, this system allows users to easily create and place desired objects and effectively interact with them in the virtual environment.

[0399] (Application example 2)

[0400] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0401] Unlike real stores, creating product displays and layouts in virtual stores can be non-intuitive and require specialized skills, so there is a demand for technology that allows non-experts to easily create and arrange displays. Furthermore, there is a lack of means to optimize interactions by reflecting human emotions, which can degrade the quality of the user experience. Therefore, there is a demand for a system that can easily and effectively generate objects in virtual spaces and optimize interactions through emotion recognition.

[0402] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for a user to specify a desired object using a prompt or an image; means for a generation AI to generate a Python script based on the specified prompt or image; means for creating an object using 3D modeling software using the generated Python script; means for placing the created object in a virtual space; emotion recognition means for analyzing the user's emotions in real time; and means for adjusting the responses of NPCs based on the emotion recognition results. This allows even non-experts to intuitively generate and place objects in a virtual space, and further optimizes interactions according to the user's emotions, enabling a high-quality user experience.

[0403] A "prompt" is text or a command that a user enters into a computer system to instruct or request something.

[0404] An "image" is digital data representing visual information that can be displayed and processed on a computer.

[0405] "Generative AI" refers to artificial intelligence technology that automatically generates new data (in this case, Python scripts) based on input data.

[0406] A "Python script" is a series of instructions or procedures written in the Python programming language.

[0407] "3D modeling software" refers to software for creating and editing three-dimensional objects and scenes on a computer.

[0408] A "virtual space" is a computer-generated digital environment or simulation.

[0409] "Emotion recognition means" refers to technology or systems that analyze a user's facial expressions and voice data to identify their emotional state.

[0410] "NPC" stands for "Non-Player-Character" and refers to a virtual character that is not controlled by the user but can be interacted with.

[0411] MODE FOR CARRYING OUT THE INVENTION

[0412] The following describes an embodiment of the present invention. The system of the present invention allows a user to specify a desired object using a prompt or an image, automatically generates a Python script using generative AI, creates the object using 3D modeling software, and places the object in a virtual space. Furthermore, an emotion engine that recognizes the user's emotions is used to optimize interactions within the virtual space.

[0413] Hardware and software used

[0414] The system is implemented using the following hardware and software:

[0415] Server: Hosts the generative AI model and emotion recognition engine, and processes prompts and image data.

[0416] User's device: This includes smart glasses, smartphones, head-mounted displays, etc., used by users to access the virtual space.

[0417] Generative AI model: Generates Python scripts based on user prompts and images.

[0418] 3D modeling software: Execute the script generated using Blender or similar software to generate a 3D model.

[0419] Emotion recognition engine: Analyzes the user's facial expressions and voice data in real time to provide appropriate interactions.

[0420] Data processing and calculation

[0421] The server receives prompts and image data entered by the user and sends them to the generative AI model, which then generates a Python script based on the input and sends it back to the server. The server then sends the Python script to the user's device, which then runs the script in the 3D modeling software.

[0422] The generated 3D model is placed in the virtual space on the user's device. The device is also equipped with an emotion recognition engine that analyzes the user's emotional data in real time. Based on this emotional data, the server adjusts the responses and behavior of the NPC (virtual character) to provide a satisfying interaction for the user.

[0423] Specific examples

[0424] For example, a user puts on smart glasses and accesses a virtual store management screen, selects an empty display shelf in the store, and enters the prompt to "create a new display" or uploads an image of an existing display.

[0425] The server receives user input and sends prompts and image data to the generative AI model, which generates a suitable Python script in response to the prompt "Create a new display." This script is then sent from the server to the user's smart glasses, where it is generated and placed as a 3D object on a virtual display within the glasses.

[0426] The user checks the generated display in the virtual store and fine-tunes its placement as needed. The emotion recognition engine reads the user's facial expressions and voice to analyze their emotional state. For example, if the user shows a satisfied expression, the emotion engine recognizes this and notifies the user that the generation process was successful.

[0427] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0428] Step 1:

[0429] Users access the virtual store management screen using devices such as smart glasses or smartphones, where they can enter a prompt to create a desired object (e.g., "Create a new display") or upload an image of an existing display. The prompt or image data is provided as input.

[0430] Step 2:

[0431] The server receives prompts and image data entered by the user and sends them to the generative AI model. Specifically, the prompts and image data are used as input, and the generative AI model analyzes them. Here, preprocessing and tokenization of the data are performed.

[0432] Step 3:

[0433] The generative AI model generates a Python script based on the received prompt and image data. The generated Python script is output. The generative AI model automatically creates a script based on the specified prompt.

[0434] Step 4:

[0435] The server sends the generated Python script to the user's terminal. This is the process by which the generated script is transferred from the server to the user's terminal. The output from the server is a Python script, and the input to the terminal is the same script.

[0436] Step 5:

[0437] The user's device executes the received Python script in 3D modeling software (such as Blender) to generate a 3D object. In this step, the script is actually executed and a 3D model is generated. The input is the Python script, and the output is the 3D model.

[0438] Step 6:

[0439] The user's device places the generated 3D model in a virtual space. Specifically, the 3D model is placed in a virtual store as a virtual display. The input is the 3D model, and the output is placement in the virtual space.

[0440] Step 7:

[0441] The emotion recognition engine installed in the device analyzes the user's facial expressions and voice in real time to recognize their emotional state. The input is the user's facial expression data and voice data, and the emotional state is recognized as data calculation. Emotional data based on the analysis is output.

[0442] Step 8:

[0443] The server adjusts the NPC's responses and behavior based on the emotional data obtained from the emotion recognition engine. The input is the output data of the emotion recognition engine, and the server processes the data to adjust the NPC's responses and behavior. The adjusted NPC settings are obtained as the output.

[0444] Step 9:

[0445] The user interacts with the generated displays and NPCs in the virtual space and fine-tunes their placement as needed. This step optimizes the user experience. Specifically, the user can view the generated displays and receive feedback based on emotion recognition. The input is the user's behavior and emotion data, and the output is an optimized user experience.

[0446] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0447] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0448] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0449] [Second embodiment]

[0450] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0451] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0452] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0453] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0454] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0455] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0456] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0457] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0458] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0459] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0460] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0461] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0462] The system of the present invention allows the user to specify the desired object using prompts or images, automatically generates a Python script using generation AI, creates the object using 3D modeling software, and places the object in virtual space.

[0463] Object Creation

[0464] First, a user accesses the Metaverse administration screen and selects the object creation option. The user enters a prompt for the desired object's characteristics and shape, or uploads an existing image. For example, a user might enter the prompt "wooden bench."

[0465] The server then receives the user's input and provides prompts and image data to the generation AI, which then creates a Python script to generate the object. This Python script, when executed in 3D modeling software, contains the specific steps and parameters required to generate the specified object.

[0466] The Python script generated by the generative AI is sent from the server to the terminal, and the terminal generates an object using 3D modeling software based on this script. Common 3D modeling software such as Blender can be used here.

[0467] Object placement in virtual space

[0468] The device also performs the procedure to place the generated object in the virtual space as a 3D model. This allows users to intuitively place the objects they have created in the virtual space and share them with other users. For example, a generated "wooden bench" can be placed in the lobby of a virtual office.

[0469] Realization of in-house office

[0470] The server then uses the AI ​​model to generate non-player characters (NPCs). These NPCs have various roles, such as acting as advisors in meetings or as idea-boosting assistants. The generated NPC data is sent from the server to the device, which can then use it to place the NPCs in the virtual space.

[0471] User participation and interaction

[0472] Finally, users participate in the virtual space as avatars. Users can select their own avatars, log in, and interact with generated objects and NPCs. In particular, NPCs not only interact with users, but also with each other, providing new ideas and recreating the lively atmosphere of an office.

[0473] Specific examples

[0474] For example, a user selects "Object Creation" on the Metaverse management screen and enters the prompt "wooden bench." The server sends this prompt to the generation AI, which generates a Python script. The server then sends the script to the device, which then runs the script in Blender to generate a 3D model of the "wooden bench." The generated model is placed in the office lobby in the virtual space.

[0475] Meanwhile, the server uses an AI model to generate an NPC that acts as an "idea-boosting assistant" and sends that data to the device. The device places this NPC in the virtual space, and the user logs in as an avatar and interacts with the NPC, allowing the idea-boosting process to take place in the virtual space.

[0476] As described above, the system of the present invention enables users to easily generate and place 3D objects and recreate the experience of a real-world office in a virtual space.

[0477] The processing flow will be explained below.

[0478] Object creation process flow

[0479] Step 1:

[0480] A user accesses the Metaverse administration screen and selects "Create Object," where they are prompted to enter the desired object's characteristics and shape, or upload an existing image.

[0481] Step 2:

[0482] The server receives input data (prompts or images) from the user and sends it to the generation AI, which then automatically creates a Python script to generate an object based on the specified prompt or image.

[0483] Step 3:

[0484] The server sends the generated Python script to the user's device, which receives it and saves it to its local disk.

[0485] Step 4:

[0486] The terminal runs the saved Python script in 3D modeling software (e.g., Blender), which generates a 3D object based on the specified properties and shape.

[0487] Object placement in virtual space

[0488] Step 5:

[0489] The device performs a procedure for placing the generated 3D object in a virtual space, specifically, exporting or placing the object generated using the 3D modeling software at a specified location in the virtual space.

[0490] Realization of in-house office

[0491] Step 1:

[0492] The server uses the AI ​​model to generate an NPC with the role specified by the user (for example, an idea-boosting assistant). This NPC has the data necessary to interact with the user and communicate with other NPCs.

[0493] Step 2:

[0494] The server sends the generated NPC data to the user's device, which receives the data and places it in the virtual space.

[0495] Step 3:

[0496] Once the device has completed placing the NPC, the NPC becomes active in the virtual space, allowing for conversation and interaction with the user and other NPCs.

[0497] User participation and interaction

[0498] Step 1:

[0499] A user logs into a virtual space as an avatar. The user selects their avatar and completes the login process.

[0500] Step 2:

[0501] The user can start interacting with the generated objects and NPCs. For example, they can provide ideas to NPCs placed as idea-boosting assistants and receive advice or new input in response.

[0502] As described above, the system of the present invention realizes a series of processes that allow users to easily generate and place 3D objects and recreate a real-world office experience in a virtual space.

[0503] Example 1

[0504] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0505] Conventional systems for generating 3D objects and placing them in virtual spaces are difficult to use unless the user has a high level of expertise. Furthermore, the interaction functions with generated objects and NPCs (Non-Player-Characters) are insufficient, resulting in poor user experience and reduced usability. Furthermore, security and processing speed issues frequently arise when exchanging generated data.

[0506] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0507] In this invention, the server includes an input means for a user to input a prompt or an image, a means for a generative AI model to generate a Python script based on the input prompt or image, and a means for creating an object with 3D modeling software using the generated Python script, which enables even users without specialized knowledge to easily generate 3D objects and intuitively place them in a virtual space.

[0508] In this invention, the server further includes a means for allowing users to participate in the virtual space as avatars, a means for generating and placing NPCs (Non-Player-Characters) in the virtual space and for the NPCs to interact with the user, and a means for the user to refer to information generated by the interactions between NPCs, thereby enabling richer interactions in the virtual space and improving the user experience.

[0509] This invention also includes a means for using a combination of an image generation algorithm and a 3D modeling algorithm to generate an object using a generative AI model, a means for transmitting the generated 3D model to a user's terminal via a server, and a means for placing the transmitted 3D model in a virtual space on the user's terminal, thereby enabling the generated data to be exchanged efficiently and safely and quickly placed in the virtual space.

[0510] "prompt"

[0511] is an input format in which a user specifies the desired properties and shape of an object in text format.

[0512] "image"

[0513] is visual data that a user uploads to visually specify an existing object.

[0514] "Generative AI model"

[0515] is a system that includes Python scripts and artificial intelligence algorithms for generating 3D objects based on user input data.

[0516] "Python script"

[0517] is program code that is automatically generated by a generative AI model and contains instructions and parameters for creating an object in 3D modeling software.

[0518] "3D modeling software"

[0519] is a software application that runs Python scripts to create and edit 3D objects. Examples include Blender.

[0520] "Object"

[0521] is a digital model with three-dimensional shape and structure that is generated to be placed in a virtual space.

[0522] "Virtual Space"

[0523] is a computer-generated environment built for users to interact with digital objects and NPCs.

[0524] "Users"

[0525] A player is a person who uses the system to input prompts or images to interact with generated objects and NPCs.

[0526] "Input means"

[0527] is the interface through which the user provides prompts and images to the system.

[0528] "Generation means"

[0529] is a component that performs processing to generate an object based on the input prompt and image.

[0530] "Method of Creation"

[0531] is a method for running the generated Python script to create the object in 3D modeling software.

[0532] "Arrangement means"

[0533] is a tool or process for properly positioning generated objects in a virtual space.

[0534] "Avatar"

[0535] is a digital character that users use to represent themselves in a virtual space.

[0536] "NPC (Non-Player-Character)"

[0537] is a digital character generated by the system that interacts with the user in a virtual space.

[0538] "Image generation algorithm"

[0539] is an algorithm that manages the process of generating objects based on user-provided image data.

[0540] "3D modeling algorithm"

[0541] is an algorithm for calculating and generating 3D shapes used by generative AI models.

[0542] "Transmission means"

[0543] is a protocol and function for transferring generated 3D models and data from a server to a user's device.

[0544] The system of the present invention allows the user to specify the desired object using prompts or images, automatically generates a Python script using a generative AI model, creates the object using 3D modeling software, and places the object in a virtual space.

[0545] Hardware and Software Configuration

[0546] Server: A high-performance server machine is used to run the generative AI model and process the data.

[0547] Terminal: Use a personal computer to accept user input, run generated Python scripts, and place 3D objects in the virtual space.

[0548] Generative AI model: An artificial intelligence model for generating Python scripts based on user prompts and images. For example, we use a natural language processing model (e.g., ChatGPT).

[0549] 3D modeling software: Software that runs Python scripts to create and edit 3D objects. Blender will be used as an example.

[0550] Object creation procedure

[0551] 1. A user accesses the Metaverse Admin screen and selects the object creation option. Here, the user enters the desired object characteristics and shape as a prompt, or uploads an existing image. In this example, the user enters the prompt "wooden bench."

[0552] 2. The server receives the user's input data (prompts and images) and provides the prompts and image data to the generative AI model. The generative AI model then creates a Python script to generate the object. This Python script contains the specific steps and parameters to be executed in the 3D modeling software.

[0553] 3. The generated Python script is sent from the server to the terminal, which then launches 3D modeling software such as Blender and executes the Python script to generate the specified object.

[0554] Object placement in virtual space

[0555] The device then performs the procedure to place the generated 3D object in the virtual space. Specifically, the user can intuitively drag and drop the object within the virtual space to place it. For example, placing a generated "wooden bench" in the lobby of a virtual office corresponds to this operation.

[0556] Realization of in-house office

[0557] The server uses AI models to generate non-player characters (NPCs). These NPCs have various roles, such as acting as advisors in meetings or as idea-boosting assistants. The generated NPC data is then sent from the server to the device, which can then use it to place the NPCs in the virtual space.

[0558] User participation and interaction

[0559] Users participate in the virtual space as avatars. They can select their own avatar, log in, and interact with generated objects and NPCs. In particular, NPCs not only interact with users, but also with each other, providing new ideas and recreating the lively atmosphere of an office.

[0560] Specific examples

[0561] As a specific example, a case will be described in which a user selects "Create Object" on the metaverse management screen and inputs the prompt "Wooden Bench."

[0562] 1. The server sends this prompt to the generation AI, which generates a Python script.

[0563] 2. The generated script is sent from the server to the device, and the device executes the script in Blender to generate a 3D model of a "wooden bench."

[0564] 3. The user drags and drops the model into the virtual office lobby.

[0565] As another specific example, we will also explain a case where the server uses an AI model to generate an NPC with the role of an "idea bouncing assistant."

[0566] 1. The generated NPC data is sent from the server to the device.

[0567] 2. The device places this NPC in the virtual space, and the user logs in as an avatar and interacts with the NPC, bouncing ideas off each other in the virtual space.

[0568] As described above, the system of the present invention allows even users without specialized knowledge to easily generate 3D objects and intuitively place them in a virtual space.

[0569] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0570] Step 1:

[0571] The user accesses the metaverse management screen and selects the option for creating the object they desire.

[0572] Input: User access and object creation choices

[0573] Output: Object creation option selection

[0574] Step 2:

[0575] The user enters the desired object characteristics or shape as a prompt, or uploads an existing image. For example, enter the prompt "wooden bench."

[0576] Input: prompt or image data

[0577] Output: Prompt or image data is reflected in the Metaverse management screen

[0578] Step 3:

[0579] The server receives the user's input data (prompts and images).

[0580] Input: prompt or image data

[0581] Output: State that receives prompt or image data

[0582] Step 4:

[0583] The server provides prompts and image data to the generated AI model.

[0584] Input: prompt or image data

[0585] Output: Providing data to a generative AI model

[0586] Step 5:

[0587] Create a Python script that will allow the generative AI model to generate objects based on the provided prompts and image data, with specific steps and parameters to be executed in the 3D modeling software.

[0588] Input: prompt or image data

[0589] Output: Python script

[0590] Step 6:

[0591] The server sends the generated Python script to the device.

[0592] Input: Python script

[0593] Output: Script sending status to the terminal

[0594] Step 7:

[0595] Based on the Python script received by the device, 3D modeling software such as Blender is launched.

[0596] Input: Python script

[0597] Output: 3D modeling software running

[0598] Step 8:

[0599] The terminal will run the Python script in Blender and create the specified object.

[0600] Input: Python script

[0601] Output: Generated 3D object

[0602] Step 9:

[0603] The device provides an interface for placing the generated 3D objects in the virtual space. Users can intuitively drag and drop objects to place them in the virtual space. For example, placing a wooden bench in the lobby of a virtual office.

[0604] Input: Generated 3D object

[0605] Output: Objects placed in virtual space

[0606] Step 10:

[0607] The server generates NPCs (Non-Player-Characters) using an AI model and sends the data to the device.

[0608] Input: None (internal data)

[0609] Output: Send NPC data to the terminal

[0610] Step 11:

[0611] The device places NPCs generated by an AI model into the virtual space.

[0612] Input: NPC data

[0613] Output: NPC placed in virtual space

[0614] Step 12:

[0615] Users participate in the virtual space as avatars. Users select their own avatar and log in.

[0616] Input: Avatar selection and login information

[0617] Output: Avatar participation status in virtual space

[0618] Step 13:

[0619] Users interact with generated objects and NPCs through conversations. By observing the conversations and information sharing between NPCs, new ideas can be gained.

[0620] Input: User actions and interactions

[0621] Output: Dialogue results and new ideas

[0622] (Application example 1)

[0623] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0624] In modern virtual spaces and virtual stores, it is difficult for users to intuitively customize desired products and objects and then view and operate the generated objects. In particular, there is a demand for a system that allows users to easily generate specific objects without specialized knowledge and view and operate them in a virtual space. Another challenge is the placement of automatic characters to enhance user interaction and provide a more realistic experience, as well as the utilization of information generated by these characters.

[0625] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0626] In this invention, the server includes a means for a user to specify a desired object using a prompt or image, a means for a generation AI to generate a program based on the specified prompt or image, and a means for creating an object using 3D modeling software using the generated program. This allows users to easily generate desired objects without any special knowledge, place them in a virtual space, and view and manipulate them. Furthermore, to enhance the user's experience, automated characters can be placed and information generated through interactions between the user and the characters can be provided.

[0627] A "prompt" is a text input that allows a user to specify the desired properties or shape of an object.

[0628] An "image" is visual data uploaded by a user to specify the properties and shape of a desired object.

[0629] A "program" is a script automatically generated by generative AI to create an object in 3D modeling software.

[0630] "3D modeling software" refers to software tools for programmatically generating 3D objects.

[0631] A "virtual space" is a computer-generated space that users can interact with.

[0632] A "user" is an entity that uses the system to specify and create a desired object.

[0633] A "terminal" is a digital device used by a user, such as a computer or smartphone.

[0634] "Automatic characters" refer to non-player characters (NPCs) that can interact with users in a virtual space and with each other.

[0635] "Generative AI" is an artificial intelligence technology that automatically creates programs to generate objects based on input prompts and images.

[0636] An "algorithm" is a step-by-step procedure or formula for solving a particular problem.

[0637] The "information processing device" is a network-compatible device for transmitting the generated three-dimensional model to the user's terminal.

[0638] A "model" is generated three-dimensional object data, and refers to a specific design that is placed in a virtual space.

[0639] An "object" refers to any object or item designated by the user, and is a target object generated in a virtual space.

[0640] The embodiment of the present invention is based on a system comprising the following steps.

[0641] First, the user accesses the virtual environment management screen using a smartphone or personal computer. This management screen provides multiple options for creating the object the user desires. The user specifies the desired object by entering a prompt or uploading an image. Examples of prompts include a "wooden bench" or a "stylish glass table."

[0642] The user's input data is then sent to a server, which uses generative AI to automatically generate a program to generate a concrete object from the prompt or image. This program is written as a Python script and can be executed in 3D modeling software (e.g., Blender).

[0643] The generated Python script is sent to the user's terminal via the network, where it is executed by the 3D modeling software to generate a specific object.

[0644] The generated 3D model is then placed directly in a virtual space on the device, allowing the user to visually confirm the object of their choice within the virtual space. This virtual space can be a virtual store, and the user can place the generated object in real life for use in sales promotions, interior design simulations, and more.

[0645] Furthermore, the server can generate automated characters and place them in the virtual space. This allows users to interact with the automated characters, resulting in a more realistic experience. The automated characters can also interact with each other, and users can view the content of their conversations and the information generated.

[0646] Hardware and software used

[0647] Server: A computer server for running generative AI

[0648] Device: A smartphone or personal computer operated by a user

[0649] 3D modeling software: Blender, etc.

[0650] Generative AI: Artificial intelligence technology that generates programs based on prompts and images

[0651] Specific examples

[0652] 1. The user selects "Object Creation" on the virtual space management screen and enters the prompt "Wooden bench."

[0653] 2. The server sends this prompt to the generation AI, which generates a Python script.

[0654] 3. The generated script is sent from the server to the device, and the device executes the script in Blender to generate a three-dimensional model of the "wooden bench."

[0655] 4. The generated model is placed in the virtual store showroom.

[0656] In this way, the present invention enables users to easily create and place desired objects and freely manipulate them in virtual space without any special knowledge.

[0657] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0658] Step 1:

[0659] A user accesses the virtual space management screen and selects the "Object Creation" option. The user is prompted to enter the characteristics and shape of an object, such as a "wooden bench," or upload an existing image.

[0660] Input: User input (prompt or image)

[0661] Output: Input prompt and image data

[0662] Step 2:

[0663] The server receives user input and sends the data to the Generative AI, which generates a Python script to generate the object based on the prompt and image.

[0664] Input: User input (prompt or image)

[0665] Data processing: Calculation process to generate Python scripts based on prompt statements

[0666] Output: Generated Python script

[0667] Step 3:

[0668] The generated Python script is sent from the server to the user's terminal.

[0669] Input: Generated Python script

[0670] Output: Python script sent to user's terminal

[0671] Step 4:

[0672] The terminal executes the received Python script in 3D modeling software (e.g., Blender) to generate an object.

[0673] Input: The received Python script

[0674] Data processing: 3D model generation processing using 3D modeling software

[0675] Output: Generated 3D model

[0676] Step 5:

[0677] The generated three-dimensional model is placed in a virtual space on the terminal, and the user can visually check and operate it.

[0678] Input: Generated 3D model

[0679] Output: 3D model placed in the virtual space on the device

[0680] Step 6:

[0681] The server uses a generation AI to generate automated characters and place them in the virtual space, allowing users to interact with them.

[0682] Input: Automatic character generation request

[0683] Data processing: AI calculation processing for automatic character generation

[0684] Output: Generated auto character

[0685] Step 7:

[0686] The automated characters converse with the user and with each other, and the content of the conversation can be viewed by the user.

[0687] Input: User interaction with automated character

[0688] Output: Interactions and generated information

[0689] Step 8:

[0690] The generated information is sent to the user's terminal, where the user can check the details.

[0691] Input: Generated information

[0692] Output: Display information on the user's terminal

[0693] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0694] The system of the present invention allows users to specify the desired object using prompts or images, automatically generates a Python script using a generation AI, creates the object using 3D modeling software, and places the object in a virtual space. Furthermore, it combines an emotion engine that recognizes the user's emotions to optimize interactions within the virtual space.

[0695] Object Creation

[0696] First, a user accesses the Metaverse administration screen and selects the object creation option. The user enters a prompt for the desired object's characteristics and shape, or uploads an existing image. For example, a user might enter the prompt "wooden bench."

[0697] The server then receives the user's input and provides prompts and image data to the generation AI, which then creates a Python script to generate the object. This Python script, when executed in 3D modeling software, contains the specific steps and parameters required to generate the specified object.

[0698] The Python script generated by the generative AI is sent from the server to the terminal, and the terminal generates an object using 3D modeling software based on this script. Common 3D modeling software such as Blender can be used here.

[0699] Object placement in virtual space

[0700] The device also performs the procedure to place the generated object in the virtual space as a 3D model. This allows users to intuitively place the objects they have created in the virtual space and share them with other users. For example, a generated "wooden bench" can be placed in the lobby of a virtual office.

[0701] Realization of in-house office

[0702] The server then uses the AI ​​model to generate non-player characters (NPCs). These NPCs have various roles, such as acting as meeting advisors or idea-boosting assistants. The generated NPC data is sent from the server to the device, which can then use it to place the NPCs in the virtual space.

[0703] User participation and interaction

[0704] Users participate in the virtual space as avatars. They can select their own avatar, log in, and interact with generated objects and NPCs. In particular, NPCs not only interact with users, but also with each other, providing new ideas and recreating the lively atmosphere of an office.

[0705] Applying the Emotion Engine

[0706] The system of the present invention further incorporates an emotion engine that recognizes the user's emotions. The emotion engine analyzes the user's voice data and facial expression data in real time to identify the user's emotional state. For example, if the user looks happy, the emotion engine will recognize this as "joy."

[0707] Based on the analysis results, the server adjusts the NPC's responses and behavior. For example, if the user looks dissatisfied, the NPC will respond by offering words of encouragement or suggesting solutions. The emotion engine's analysis results are also reflected in the dialogue between NPCs, resulting in more plausible interactions.

[0708] Specific examples

[0709] For example, a user selects "Object Creation" on the Metaverse management screen and enters the prompt "wooden bench." The server sends this prompt to the generation AI, which generates a Python script. The server then sends the script to the device, which then runs the script in Blender to generate a 3D model of the "wooden bench." The generated model is placed in the office lobby in the virtual space.

[0710] Meanwhile, the server uses an AI model to generate an NPC that acts as an "idea-boosting assistant" and sends that data to the device. The device places this NPC in the virtual space, and the user logs in as an avatar and interacts with the NPC, allowing the idea-boosting process to take place in the virtual space.

[0711] Furthermore, if the user expresses dissatisfaction during the interaction, the emotion engine will analyze and recognize this information in real time, and the server will adjust the NPC's response accordingly, providing specific advice and solutions to the user's problems. In this way, the emotion engine optimizes the interaction and improves the user experience.

[0712] As described above, the system of the present invention allows users to easily generate and place 3D objects, realizing a series of processes that recreate a real-world office experience in a virtual space. Furthermore, by combining it with an emotion engine, it is possible to optimize interactions and increase user satisfaction.

[0713] The processing flow will be explained below.

[0714] Object creation process flow

[0715] Step 1:

[0716] A user accesses the Metaverse administration screen and selects "Create Object." They are prompted to enter the desired object's characteristics and shape, or they can upload an existing image.

[0717] Step 2:

[0718] The server receives input data (prompts or images) from the user and sends it to the generation AI, which then automatically creates a Python script to generate an object based on the specified prompt or image.

[0719] Step 3:

[0720] The server sends the generated Python script to the user's device, which receives it and saves it to its local disk.

[0721] Step 4:

[0722] The terminal runs the saved Python script in 3D modeling software (e.g., Blender), which generates a 3D object based on the specified properties and shape.

[0723] Processing flow for placing objects in virtual space

[0724] Step 5:

[0725] The device performs a procedure for placing the generated 3D object in a virtual space, specifically, exporting or placing the object generated using the 3D modeling software at a specified location in the virtual space.

[0726] Process flow for realizing in-house office

[0727] Step 1:

[0728] The server uses the AI ​​model to generate an NPC with the role specified by the user (for example, an idea-boosting assistant). This NPC has the data necessary to interact with the user and communicate with other NPCs.

[0729] Step 2:

[0730] The server sends the generated NPC data to the user's device, which receives the data and places it in the virtual space.

[0731] Step 3:

[0732] Once the device has completed placing the NPC, the NPC becomes active in the virtual space, allowing for conversation and interaction with the user and other NPCs.

[0733] User participation and interaction process flow

[0734] Step 1:

[0735] A user logs into a virtual space as an avatar. The user selects their avatar and completes the login process.

[0736] Step 2:

[0737] The user can start interacting with the generated objects and NPCs. For example, they can provide ideas to NPCs placed as idea-boosting assistants and receive advice or new input in response.

[0738] Process flow for applying the emotion engine

[0739] Step 1:

[0740] When a user interacts in a virtual space, the emotion engine collects the user's voice data and facial expression data in real time, analyzes this data, and identifies the user's emotional state.

[0741] Step 2:

[0742] The server receives the analysis results of the emotion engine and adjusts the NPC's response. For example, if the user has a dissatisfied expression, the NPC will respond by offering words of encouragement or suggesting a solution.

[0743] Step 3:

[0744] The server adjusts the dialogue between other NPCs based on the results of the emotion engine, enabling more natural communication within the virtual space.

[0745] Specific examples

[0746] A concrete example of object creation

[0747] For example, a user selects "Object Creation" on the Metaverse management screen and enters the prompt "wooden bench." The server sends this prompt to the generation AI, which generates a Python script. The server then sends the script to the device, which then runs the script in Blender to generate a 3D model of the "wooden bench." The generated model is placed in the office lobby in the virtual space.

[0748] Example of NPC generation and dialogue

[0749] The server uses an AI model to generate an NPC that acts as an "idea-boosting assistant" and sends that data to the device. The device places this NPC in the virtual space, and the user logs in as an avatar and interacts with the NPC, allowing the idea-boosting process to take place in the virtual space.

[0750] Specific examples of emotion engine application

[0751] If the user expresses dissatisfaction during the interaction, the emotion engine will analyze and recognize this information in real time, and the server will adjust the NPC's response accordingly, providing specific advice and solutions to the user's problems. In this way, the emotion engine optimizes the interaction and improves the user experience.

[0752] As described above, the system of the present invention allows users to easily generate and place 3D objects, realizing a series of processes that recreate a real-world office experience in a virtual space. Furthermore, by combining it with an emotion engine, the user experience can be further improved.

[0753] Example 2

[0754] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0755] Interaction in modern virtual environments requires a method that allows users to easily create and place objects. However, existing systems require specialized knowledge and numerous manual operations, and few are easy to use. Furthermore, there is a lack of means to accurately reflect the user's emotions and achieve more natural and effective interactions in virtual environments, limiting the quality of the user experience.

[0756] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0757] In this invention, the server includes: a prompt or image input means for a user to specify a desired object; a generation AI generating code based on the specified prompt or image; a 3D model creation software creating an object using the generated code; a virtual environment placing the created object; and a sentiment analysis engine recognizing the user's sentiment and adjusting the responses and behavior of a virtual character based on the recognition results. This allows users to easily create and place 3D objects without requiring specialized knowledge, and enables natural and effective dialogue within the virtual environment utilizing sentiment analysis.

[0758] A "prompt" is input data that describes the target or condition desired by the user in natural language.

[0759] An "image" is digital data that visually represents an object or condition specified by a user.

[0760] "Generative AI" is an artificial intelligence system that analyzes user input data and automatically generates the necessary code and models.

[0761] "Code" refers to the program instructions generated by the AI ​​based on prompts and images specified by the user.

[0762] "3D model creation software" is software that actually generates 3D objects based on generated code.

[0763] A "virtual environment" is a computer-generated simulated space in which a user can interact.

[0764] A "virtual character" is an NPC (non-player character) in a virtual environment, a character that is set up to converse and interact with the user.

[0765] An "emotion analysis engine" is a technology that analyzes a user's voice, facial expressions, etc. to identify their emotional state.

[0766] "User's computer" means an information processing device used by a user to receive data from the generated AI and use it within the virtual environment.

[0767] An "information processing device" is a device that transmits, receives, analyzes, and stores digital data, and examples of this include servers.

[0768] The present invention provides a system that allows a user to easily create desired objects and place them in a virtual environment. Specific embodiments of the system will be described below.

[0769] Hardware and software used

[0770] This system mainly uses the following hardware and software:

[0771] Server: Receives prompts and image data and sends them to the generation AI

[0772] Terminal: A computer operated by a user, with 3D modeling software such as Blender installed.

[0773] Generative AI model: Analyzes user input data and generates the necessary Python scripts

[0774] Virtual environment management software: Place and manage objects and virtual characters in a virtual environment

[0775] Emotion Analysis Engine: Analyzes user emotions and adjusts the virtual character's responses and behavior

[0776] Detailed System Description

[0777] 1. Object creation prompt input

[0778] A user opens a browser and logs into the Metaverse admin page, where they select the "Create Object" option and enter the desired object's characteristics and shape as a prompt, or upload an existing image (e.g., "Wooden Bench").

[0779] Example prompt sentence:

[0780] "Generate a wooden bench."

[0781] "Create a virtual character that acts as a meeting advisor."

[0782] 2. Generate prompts and send them to the AI

[0783] The server receives user input and provides prompts and image data to the generative AI, which then creates a Python script to generate the object. This script contains commands and parameters that can be executed in 3D modeling software such as Blender.

[0784] 3. Generate and run the Python script

[0785] The AI ​​analyzes the prompt and automatically generates a Python script, which is then sent to the user's device via a server. The device then uses 3D modeling software such as Blender to generate objects based on the script.

[0786] 4. Placing objects in the virtual environment

[0787] The device places the generated 3D model in the virtual environment, and the user can adjust the object's exact position using the mouse or keyboard. Information about the placed object is registered in the virtual environment management software and shared with other users.

[0788] 5. Virtual Character Generation and Emotional Response

[0789] The server uses AI models to generate virtual characters, which can act as meeting advisors, idea-boosting assistants, etc. These characters are sent to the device based on the data generated by the AI ​​and placed in the virtual environment.

[0790] In addition, an emotion analysis engine analyzes the user's voice and facial expressions in real time and adjusts the virtual character's responses and behavior accordingly. For example, if the user looks unhappy, the virtual character will offer words of encouragement or suggest solutions to the problem.

[0791] Specific examples

[0792] For example, consider the case of generating a "wooden bench." The user selects "Object Creation" on the metaverse management screen and enters "wooden bench" as a prompt. The server sends this prompt to the generation AI, which creates a Python script. The server then sends the script to the device, which then runs it in Blender. The resulting 3D model of the "wooden bench" is placed in the lobby of the virtual office.

[0793] In addition, a virtual character called a "meeting advisor" is generated, and when a user interacts with the virtual environment, an emotion analysis engine analyzes the user's emotional state in real time and adjusts the virtual character's responses accordingly, allowing users to experience more natural and effective interactions within the virtual environment.

[0794] The above is a concrete example of how to implement the invention. This system allows users to easily generate 3D objects without requiring specialized knowledge, and enables effective dialogue utilizing emotion analysis.

[0795] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0796] Step 1:

[0797] The user opens a browser and accesses the metaverse management screen. The login screen is displayed, the user enters their username and password, and clicks the login button. The entered username and password are sent to the server for authentication. If authentication is successful, the metaverse management screen is displayed.

[0798] Input: Username, Password

[0799] Output: Metaverse admin page

[0800] Step 2:

[0801] On the Metaverse Admin screen, select the "Create Object" option. The user enters the desired object's characteristics and shape as a prompt, or uploads an existing image. For example, enter the prompt "Wooden bench." The user's input is sent to the server.

[0802] Input: A prompt or image (e.g., "Wooden bench")

[0803] Output: Prompt data sent to the server

[0804] Step 3:

[0805] The server takes the user's input and sends it to a generative AI, which parses the prompt and generates a Python script containing commands and parameters that can be executed in 3D modeling software such as Blender.

[0806] Input: prompt data

[0807] Output: Python script

[0808] Step 4:

[0809] The generated Python script is sent from the server to the device, where it is executed in 3D modeling software such as Blender. The specified object is generated based on the script.

[0810] Input: Python script

[0811] Output: 3D object

[0812] Step 5:

[0813] The device places the generated 3D object in the virtual environment. The user can adjust the object's position using the mouse or keyboard. The information about the placed object is saved in the virtual environment management software and can be shared with other users.

[0814] Input: 3D object

[0815] Output: Objects placed in the virtual environment

[0816] Step 6:

[0817] The server uses an AI model to generate a virtual character, which can act as a meeting advisor or idea-boosting assistant, for example. The generated character data is sent from the server to the device, which then places it in the virtual environment.

[0818] Input: Virtual character role (e.g., conference advisor)

[0819] Output: Generated virtual character data

[0820] Step 7:

[0821] The device places a virtual character in the virtual environment. When a user logs into the virtual environment as an avatar, they can begin interacting with the virtual character. An emotion analysis engine analyzes the user's emotions in real time and adjusts the virtual character's responses and behavior based on the results.

[0822] Input: Virtual character data, user emotion data

[0823] Output: User interaction with virtual character

[0824] Through the above process, this system allows users to easily create and place desired objects and effectively interact with them in the virtual environment.

[0825] (Application example 2)

[0826] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0827] Unlike real stores, creating product displays and layouts in virtual stores can be non-intuitive and require specialized skills, so there is a demand for technology that allows non-experts to easily create and arrange displays. Furthermore, there is a lack of means to optimize interactions by reflecting human emotions, which can degrade the quality of the user experience. Therefore, there is a demand for a system that can easily and effectively generate objects in virtual spaces and optimize interactions through emotion recognition.

[0828] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for a user to specify a desired object using a prompt or an image; means for a generation AI to generate a Python script based on the specified prompt or image; means for creating an object using 3D modeling software using the generated Python script; means for placing the created object in a virtual space; emotion recognition means for analyzing the user's emotions in real time; and means for adjusting the responses of NPCs based on the emotion recognition results. This allows even non-experts to intuitively generate and place objects in a virtual space, and further optimizes interactions according to the user's emotions, enabling a high-quality user experience.

[0829] A "prompt" is text or a command that a user enters into a computer system to instruct or request something.

[0830] An "image" is digital data representing visual information that can be displayed and processed on a computer.

[0831] "Generative AI" refers to artificial intelligence technology that automatically generates new data (in this case, Python scripts) based on input data.

[0832] A "Python script" is a series of instructions or procedures written in the Python programming language.

[0833] "3D modeling software" refers to software for creating and editing three-dimensional objects and scenes on a computer.

[0834] A "virtual space" is a computer-generated digital environment or simulation.

[0835] "Emotion recognition means" refers to technology or systems that analyze a user's facial expressions and voice data to identify their emotional state.

[0836] "NPC" stands for "Non-Player-Character" and refers to a virtual character that is not controlled by the user but can be interacted with.

[0837] MODE FOR CARRYING OUT THE INVENTION

[0838] The following describes an embodiment of the present invention. The system of the present invention allows a user to specify a desired object using a prompt or an image, automatically generates a Python script using generative AI, creates the object using 3D modeling software, and places the object in a virtual space. Furthermore, an emotion engine that recognizes the user's emotions is used to optimize interactions within the virtual space.

[0839] Hardware and software used

[0840] The system is implemented using the following hardware and software:

[0841] Server: Hosts the generative AI model and emotion recognition engine, and processes prompts and image data.

[0842] User's device: This includes smart glasses, smartphones, head-mounted displays, etc., used by users to access the virtual space.

[0843] Generative AI model: Generates Python scripts based on user prompts and images.

[0844] 3D modeling software: Execute the script generated using Blender or similar software to generate a 3D model.

[0845] Emotion recognition engine: Analyzes the user's facial expressions and voice data in real time to provide appropriate interactions.

[0846] Data processing and calculation

[0847] The server receives prompts and image data entered by the user and sends them to the generative AI model, which then generates a Python script based on the input and sends it back to the server. The server then sends the Python script to the user's device, which then runs the script in the 3D modeling software.

[0848] The generated 3D model is placed in the virtual space on the user's device. The device is also equipped with an emotion recognition engine that analyzes the user's emotional data in real time. Based on this emotional data, the server adjusts the responses and behavior of the NPC (virtual character) to provide a satisfying interaction for the user.

[0849] Specific examples

[0850] For example, a user puts on smart glasses and accesses a virtual store management screen, selects an empty display shelf in the store, and enters the prompt to "create a new display" or uploads an image of an existing display.

[0851] The server receives user input and sends prompts and image data to the generative AI model, which generates a suitable Python script in response to the prompt "Create a new display." This script is then sent from the server to the user's smart glasses, where it is generated and placed as a 3D object on a virtual display within the glasses.

[0852] The user checks the generated display in the virtual store and fine-tunes its placement as needed. The emotion recognition engine reads the user's facial expressions and voice to analyze their emotional state. For example, if the user shows a satisfied expression, the emotion engine recognizes this and notifies the user that the generation process was successful.

[0853] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0854] Step 1:

[0855] Users access the virtual store management screen using devices such as smart glasses or smartphones, where they can enter a prompt to create a desired object (e.g., "Create a new display") or upload an image of an existing display. The prompt or image data is provided as input.

[0856] Step 2:

[0857] The server receives prompts and image data entered by the user and sends them to the generative AI model. Specifically, the prompts and image data are used as input, and the generative AI model analyzes them. Here, preprocessing and tokenization of the data are performed.

[0858] Step 3:

[0859] The generative AI model generates a Python script based on the received prompt and image data. The generated Python script is output. The generative AI model automatically creates a script based on the specified prompt.

[0860] Step 4:

[0861] The server sends the generated Python script to the user's terminal. This is the process by which the generated script is transferred from the server to the user's terminal. The output from the server is a Python script, and the input to the terminal is the same script.

[0862] Step 5:

[0863] The user's device executes the received Python script in 3D modeling software (such as Blender) to generate a 3D object. In this step, the script is actually executed and a 3D model is generated. The input is the Python script, and the output is the 3D model.

[0864] Step 6:

[0865] The user's device places the generated 3D model in a virtual space. Specifically, the 3D model is placed in a virtual store as a virtual display. The input is the 3D model, and the output is placement in the virtual space.

[0866] Step 7:

[0867] The emotion recognition engine installed in the device analyzes the user's facial expressions and voice in real time to recognize their emotional state. The input is the user's facial expression data and voice data, and the emotional state is recognized as data calculation. Emotional data based on the analysis is output.

[0868] Step 8:

[0869] The server adjusts the NPC's responses and behavior based on the emotional data obtained from the emotion recognition engine. The input is the output data of the emotion recognition engine, and the server processes the data to adjust the NPC's responses and behavior. The adjusted NPC settings are obtained as the output.

[0870] Step 9:

[0871] The user interacts with the generated displays and NPCs in the virtual space and fine-tunes their placement as needed. This step optimizes the user experience. Specifically, the user can view the generated displays and receive feedback based on emotion recognition. The input is the user's behavior and emotion data, and the output is an optimized user experience.

[0872] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0873] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0874] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0875] [Third embodiment]

[0876] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0877] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0878] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0879] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0880] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0881] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0882] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0883] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0884] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0885] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0886] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0887] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0888] The system of the present invention allows the user to specify the desired object using prompts or images, automatically generates a Python script using generation AI, creates the object using 3D modeling software, and places the object in virtual space.

[0889] Object Creation

[0890] First, a user accesses the Metaverse administration screen and selects the object creation option. The user enters a prompt for the desired object's characteristics and shape, or uploads an existing image. For example, a user might enter the prompt "wooden bench."

[0891] The server then receives the user's input and provides prompts and image data to the generation AI, which then creates a Python script to generate the object. This Python script, when executed in 3D modeling software, contains the specific steps and parameters required to generate the specified object.

[0892] The Python script generated by the generative AI is sent from the server to the terminal, and the terminal generates an object using 3D modeling software based on this script. Common 3D modeling software such as Blender can be used here.

[0893] Object placement in virtual space

[0894] The device also performs the procedure to place the generated object in the virtual space as a 3D model. This allows users to intuitively place the objects they have created in the virtual space and share them with other users. For example, a generated "wooden bench" can be placed in the lobby of a virtual office.

[0895] Realization of in-house office

[0896] The server then uses the AI ​​model to generate non-player characters (NPCs). These NPCs have various roles, such as acting as advisors in meetings or as idea-boosting assistants. The generated NPC data is sent from the server to the device, which can then use it to place the NPCs in the virtual space.

[0897] User participation and interaction

[0898] Finally, users participate in the virtual space as avatars. Users can select their own avatars, log in, and interact with generated objects and NPCs. In particular, NPCs not only interact with users, but also with each other, providing new ideas and recreating the lively atmosphere of an office.

[0899] Specific examples

[0900] For example, a user selects "Object Creation" on the Metaverse management screen and enters the prompt "wooden bench." The server sends this prompt to the generation AI, which generates a Python script. The server then sends the script to the device, which then runs the script in Blender to generate a 3D model of the "wooden bench." The generated model is placed in the office lobby in the virtual space.

[0901] Meanwhile, the server uses an AI model to generate an NPC that acts as an "idea-boosting assistant" and sends that data to the device. The device places this NPC in the virtual space, and the user logs in as an avatar and interacts with the NPC, allowing the idea-boosting process to take place in the virtual space.

[0902] As described above, the system of the present invention enables users to easily generate and place 3D objects and recreate the experience of a real-world office in a virtual space.

[0903] The processing flow will be explained below.

[0904] Object creation process flow

[0905] Step 1:

[0906] A user accesses the Metaverse administration screen and selects "Create Object," where they are prompted to enter the desired object's characteristics and shape, or upload an existing image.

[0907] Step 2:

[0908] The server receives input data (prompts or images) from the user and sends it to the generation AI, which then automatically creates a Python script to generate an object based on the specified prompt or image.

[0909] Step 3:

[0910] The server sends the generated Python script to the user's device, which receives it and saves it to its local disk.

[0911] Step 4:

[0912] The terminal runs the saved Python script in 3D modeling software (e.g., Blender), which generates a 3D object based on the specified properties and shape.

[0913] Object placement in virtual space

[0914] Step 5:

[0915] The device performs a procedure for placing the generated 3D object in a virtual space, specifically, exporting or placing the object generated using the 3D modeling software at a specified location in the virtual space.

[0916] Realization of in-house office

[0917] Step 1:

[0918] The server uses the AI ​​model to generate an NPC with the role specified by the user (for example, an idea-boosting assistant). This NPC has the data necessary to interact with the user and communicate with other NPCs.

[0919] Step 2:

[0920] The server sends the generated NPC data to the user's device, which receives the data and places it in the virtual space.

[0921] Step 3:

[0922] Once the device has completed placing the NPC, the NPC becomes active in the virtual space, allowing for conversation and interaction with the user and other NPCs.

[0923] User participation and interaction

[0924] Step 1:

[0925] A user logs into a virtual space as an avatar. The user selects their avatar and completes the login process.

[0926] Step 2:

[0927] The user can start interacting with the generated objects and NPCs. For example, they can provide ideas to NPCs placed as idea-boosting assistants and receive advice or new input in response.

[0928] As described above, the system of the present invention realizes a series of processes that allow users to easily generate and place 3D objects and recreate a real-world office experience in a virtual space.

[0929] Example 1

[0930] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0931] Conventional systems for generating 3D objects and placing them in virtual spaces are difficult to use unless the user has a high level of expertise. Furthermore, the interaction functions with generated objects and NPCs (Non-Player-Characters) are insufficient, resulting in poor user experience and reduced usability. Furthermore, security and processing speed issues frequently arise when exchanging generated data.

[0932] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0933] In this invention, the server includes an input means for a user to input a prompt or an image, a means for a generative AI model to generate a Python script based on the input prompt or image, and a means for creating an object with 3D modeling software using the generated Python script, which enables even users without specialized knowledge to easily generate 3D objects and intuitively place them in a virtual space.

[0934] In this invention, the server further includes a means for allowing users to participate in the virtual space as avatars, a means for generating and placing NPCs (Non-Player-Characters) in the virtual space and for the NPCs to interact with the user, and a means for the user to refer to information generated by the interactions between NPCs, thereby enabling richer interactions in the virtual space and improving the user experience.

[0935] This invention also includes a means for using a combination of an image generation algorithm and a 3D modeling algorithm to generate an object using a generative AI model, a means for transmitting the generated 3D model to a user's terminal via a server, and a means for placing the transmitted 3D model in a virtual space on the user's terminal, thereby enabling the generated data to be exchanged efficiently and safely and quickly placed in the virtual space.

[0936] "prompt"

[0937] is an input format in which a user specifies the desired properties and shape of an object in text format.

[0938] "image"

[0939] is visual data that a user uploads to visually specify an existing object.

[0940] "Generative AI model"

[0941] is a system that includes Python scripts and artificial intelligence algorithms for generating 3D objects based on user input data.

[0942] "Python script"

[0943] is program code that is automatically generated by a generative AI model and contains instructions and parameters for creating an object in 3D modeling software.

[0944] "3D modeling software"

[0945] is a software application that runs Python scripts to create and edit 3D objects. Examples include Blender.

[0946] "Object"

[0947] is a digital model with three-dimensional shape and structure that is generated to be placed in a virtual space.

[0948] "Virtual Space"

[0949] is a computer-generated environment built for users to interact with digital objects and NPCs.

[0950] "Users"

[0951] A player is a person who uses the system to input prompts or images to interact with generated objects and NPCs.

[0952] "Input means"

[0953] is the interface through which the user provides prompts and images to the system.

[0954] "Generation means"

[0955] is a component that performs processing to generate an object based on the input prompt and image.

[0956] "Method of Creation"

[0957] is a method for running the generated Python script to create the object in 3D modeling software.

[0958] "Arrangement means"

[0959] is a tool or process for properly positioning generated objects in a virtual space.

[0960] "Avatar"

[0961] is a digital character that users use to represent themselves in a virtual space.

[0962] "NPC (Non-Player-Character)"

[0963] is a digital character generated by the system that interacts with the user in a virtual space.

[0964] "Image generation algorithm"

[0965] is an algorithm that manages the process of generating objects based on user-provided image data.

[0966] "3D modeling algorithm"

[0967] is an algorithm for calculating and generating 3D shapes used by generative AI models.

[0968] "Transmission means"

[0969] is a protocol and function for transferring generated 3D models and data from a server to a user's device.

[0970] The system of the present invention allows the user to specify the desired object using prompts or images, automatically generates a Python script using a generative AI model, creates the object using 3D modeling software, and places the object in a virtual space.

[0971] Hardware and Software Configuration

[0972] Server: A high-performance server machine is used to run the generative AI model and process the data.

[0973] Terminal: Use a personal computer to accept user input, run generated Python scripts, and place 3D objects in the virtual space.

[0974] Generative AI model: An artificial intelligence model for generating Python scripts based on user prompts and images. For example, we use a natural language processing model (e.g., ChatGPT).

[0975] 3D modeling software: Software that runs Python scripts to create and edit 3D objects. Blender will be used as an example.

[0976] Object creation procedure

[0977] 1. A user accesses the Metaverse Admin screen and selects the object creation option. Here, the user enters the desired object characteristics and shape as a prompt, or uploads an existing image. In this example, the user enters the prompt "wooden bench."

[0978] 2. The server receives the user's input data (prompts and images) and provides the prompts and image data to the generative AI model. The generative AI model then creates a Python script to generate the object. This Python script contains the specific steps and parameters to be executed in the 3D modeling software.

[0979] 3. The generated Python script is sent from the server to the terminal, which then launches 3D modeling software such as Blender and executes the Python script to generate the specified object.

[0980] Object placement in virtual space

[0981] The device then performs the procedure to place the generated 3D object in the virtual space. Specifically, the user can intuitively drag and drop the object within the virtual space to place it. For example, placing a generated "wooden bench" in the lobby of a virtual office corresponds to this operation.

[0982] Realization of in-house office

[0983] The server uses AI models to generate non-player characters (NPCs). These NPCs have various roles, such as acting as advisors in meetings or as idea-boosting assistants. The generated NPC data is then sent from the server to the device, which can then use it to place the NPCs in the virtual space.

[0984] User participation and interaction

[0985] Users participate in the virtual space as avatars. They can select their own avatar, log in, and interact with generated objects and NPCs. In particular, NPCs not only interact with users, but also with each other, providing new ideas and recreating the lively atmosphere of an office.

[0986] Specific examples

[0987] As a specific example, a case will be described in which a user selects "Create Object" on the metaverse management screen and inputs the prompt "Wooden Bench."

[0988] 1. The server sends this prompt to the generation AI, which generates a Python script.

[0989] 2. The generated script is sent from the server to the device, and the device executes the script in Blender to generate a 3D model of a "wooden bench."

[0990] 3. The user drags and drops the model into the virtual office lobby.

[0991] As another specific example, we will also explain a case where the server uses an AI model to generate an NPC with the role of an "idea bouncing assistant."

[0992] 1. The generated NPC data is sent from the server to the device.

[0993] 2. The device places this NPC in the virtual space, and the user logs in as an avatar and interacts with the NPC, bouncing ideas off each other in the virtual space.

[0994] As described above, the system of the present invention allows even users without specialized knowledge to easily generate 3D objects and intuitively place them in a virtual space.

[0995] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0996] Step 1:

[0997] The user accesses the metaverse management screen and selects the option for creating the object they desire.

[0998] Input: User access and object creation choices

[0999] Output: Object creation option selection

[1000] Step 2:

[1001] The user enters the desired object characteristics or shape as a prompt, or uploads an existing image. For example, enter the prompt "wooden bench."

[1002] Input: prompt or image data

[1003] Output: Prompt or image data is reflected in the Metaverse management screen

[1004] Step 3:

[1005] The server receives the user's input data (prompts and images).

[1006] Input: prompt or image data

[1007] Output: State that receives prompt or image data

[1008] Step 4:

[1009] The server provides prompts and image data to the generated AI model.

[1010] Input: prompt or image data

[1011] Output: Providing data to a generative AI model

[1012] Step 5:

[1013] Create a Python script that will allow the generative AI model to generate objects based on the provided prompts and image data, with specific steps and parameters to be executed in the 3D modeling software.

[1014] Input: prompt or image data

[1015] Output: Python script

[1016] Step 6:

[1017] The server sends the generated Python script to the device.

[1018] Input: Python script

[1019] Output: Script sending status to the terminal

[1020] Step 7:

[1021] Based on the Python script received by the device, 3D modeling software such as Blender is launched.

[1022] Input: Python script

[1023] Output: 3D modeling software running

[1024] Step 8:

[1025] The terminal will run the Python script in Blender and create the specified object.

[1026] Input: Python script

[1027] Output: Generated 3D object

[1028] Step 9:

[1029] The device provides an interface for placing the generated 3D objects in the virtual space. Users can intuitively drag and drop objects to place them in the virtual space. For example, placing a wooden bench in the lobby of a virtual office.

[1030] Input: Generated 3D object

[1031] Output: Objects placed in virtual space

[1032] Step 10:

[1033] The server generates NPCs (Non-Player-Characters) using an AI model and sends the data to the device.

[1034] Input: None (internal data)

[1035] Output: Send NPC data to the terminal

[1036] Step 11:

[1037] The device places NPCs generated by an AI model into the virtual space.

[1038] Input: NPC data

[1039] Output: NPC placed in virtual space

[1040] Step 12:

[1041] Users participate in the virtual space as avatars. Users select their own avatar and log in.

[1042] Input: Avatar selection and login information

[1043] Output: Avatar participation status in virtual space

[1044] Step 13:

[1045] Users interact with generated objects and NPCs through conversations. By observing the conversations and information sharing between NPCs, new ideas can be gained.

[1046] Input: User actions and interactions

[1047] Output: Dialogue results and new ideas

[1048] (Application example 1)

[1049] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1050] In modern virtual spaces and virtual stores, it is difficult for users to intuitively customize desired products and objects and then view and operate the generated objects. In particular, there is a demand for a system that allows users to easily generate specific objects without specialized knowledge and view and operate them in a virtual space. Another challenge is the placement of automatic characters to enhance user interaction and provide a more realistic experience, as well as the utilization of information generated by these characters.

[1051] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1052] In this invention, the server includes a means for a user to specify a desired object using a prompt or image, a means for a generation AI to generate a program based on the specified prompt or image, and a means for creating an object using 3D modeling software using the generated program. This allows users to easily generate desired objects without any special knowledge, place them in a virtual space, and view and manipulate them. Furthermore, to enhance the user's experience, automated characters can be placed and information generated through interactions between the user and the characters can be provided.

[1053] A "prompt" is a text input that allows a user to specify the desired properties or shape of an object.

[1054] An "image" is visual data uploaded by a user to specify the properties and shape of a desired object.

[1055] A "program" is a script automatically generated by generative AI to create an object in 3D modeling software.

[1056] "3D modeling software" refers to software tools for programmatically generating 3D objects.

[1057] A "virtual space" is a computer-generated space that users can interact with.

[1058] A "user" is an entity that uses the system to specify and create a desired object.

[1059] A "terminal" is a digital device used by a user, such as a computer or smartphone.

[1060] "Automatic characters" refer to non-player characters (NPCs) that can interact with users in a virtual space and with each other.

[1061] "Generative AI" is an artificial intelligence technology that automatically creates programs to generate objects based on input prompts and images.

[1062] An "algorithm" is a step-by-step procedure or formula for solving a particular problem.

[1063] The "information processing device" is a network-compatible device for transmitting the generated three-dimensional model to the user's terminal.

[1064] A "model" is generated three-dimensional object data, and refers to a specific design that is placed in a virtual space.

[1065] An "object" refers to any object or item designated by the user, and is a target object generated in a virtual space.

[1066] The embodiment of the present invention is based on a system comprising the following steps.

[1067] First, the user accesses the virtual environment management screen using a smartphone or personal computer. This management screen provides multiple options for creating the object the user desires. The user specifies the desired object by entering a prompt or uploading an image. Examples of prompts include a "wooden bench" or a "stylish glass table."

[1068] The user's input data is then sent to a server, which uses generative AI to automatically generate a program to generate a concrete object from the prompt or image. This program is written as a Python script and can be executed in 3D modeling software (e.g., Blender).

[1069] The generated Python script is sent to the user's terminal via the network, where it is executed by the 3D modeling software to generate a specific object.

[1070] The generated 3D model is then placed directly in a virtual space on the device, allowing the user to visually confirm the object of their choice within the virtual space. This virtual space can be a virtual store, and the user can place the generated object in real life for use in sales promotions, interior design simulations, and more.

[1071] Furthermore, the server can generate automated characters and place them in the virtual space. This allows users to interact with the automated characters, resulting in a more realistic experience. The automated characters can also interact with each other, and users can view the content of their conversations and the information generated.

[1072] Hardware and software used

[1073] Server: A computer server for running generative AI

[1074] Device: A smartphone or personal computer operated by a user

[1075] 3D modeling software: Blender, etc.

[1076] Generative AI: Artificial intelligence technology that generates programs based on prompts and images

[1077] Specific examples

[1078] 1. The user selects "Object Creation" on the virtual space management screen and enters the prompt "Wooden bench."

[1079] 2. The server sends this prompt to the generation AI, which generates a Python script.

[1080] 3. The generated script is sent from the server to the device, and the device executes the script in Blender to generate a three-dimensional model of the "wooden bench."

[1081] 4. The generated model is placed in the virtual store showroom.

[1082] In this way, the present invention enables users to easily create and place desired objects and freely manipulate them in virtual space without any special knowledge.

[1083] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1084] Step 1:

[1085] A user accesses the virtual space management screen and selects the "Object Creation" option. The user is prompted to enter the characteristics and shape of an object, such as a "wooden bench," or upload an existing image.

[1086] Input: User input (prompt or image)

[1087] Output: Input prompt and image data

[1088] Step 2:

[1089] The server receives user input and sends the data to the Generative AI, which generates a Python script to generate the object based on the prompt and image.

[1090] Input: User input (prompt or image)

[1091] Data processing: Calculation process to generate Python scripts based on prompt statements

[1092] Output: Generated Python script

[1093] Step 3:

[1094] The generated Python script is sent from the server to the user's terminal.

[1095] Input: Generated Python script

[1096] Output: Python script sent to user's terminal

[1097] Step 4:

[1098] The terminal executes the received Python script in 3D modeling software (e.g., Blender) to generate an object.

[1099] Input: The received Python script

[1100] Data processing: 3D model generation processing using 3D modeling software

[1101] Output: Generated 3D model

[1102] Step 5:

[1103] The generated three-dimensional model is placed in a virtual space on the terminal, and the user can visually check and operate it.

[1104] Input: Generated 3D model

[1105] Output: 3D model placed in the virtual space on the device

[1106] Step 6:

[1107] The server uses a generation AI to generate automated characters and place them in the virtual space, allowing users to interact with them.

[1108] Input: Automatic character generation request

[1109] Data processing: AI calculation processing for automatic character generation

[1110] Output: Generated auto character

[1111] Step 7:

[1112] The automated characters converse with the user and with each other, and the content of the conversation can be viewed by the user.

[1113] Input: User interaction with automated character

[1114] Output: Interactions and generated information

[1115] Step 8:

[1116] The generated information is sent to the user's terminal, where the user can check the details.

[1117] Input: Generated information

[1118] Output: Display information on the user's terminal

[1119] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1120] The system of the present invention allows users to specify the desired object using prompts or images, automatically generates a Python script using a generation AI, creates the object using 3D modeling software, and places the object in a virtual space. Furthermore, it combines an emotion engine that recognizes the user's emotions to optimize interactions within the virtual space.

[1121] Object Creation

[1122] First, a user accesses the Metaverse administration screen and selects the object creation option. The user enters a prompt for the desired object's characteristics and shape, or uploads an existing image. For example, a user might enter the prompt "wooden bench."

[1123] The server then receives the user's input and provides prompts and image data to the generation AI, which then creates a Python script to generate the object. This Python script, when executed in 3D modeling software, contains the specific steps and parameters required to generate the specified object.

[1124] The Python script generated by the generative AI is sent from the server to the terminal, and the terminal generates an object using 3D modeling software based on this script. Common 3D modeling software such as Blender can be used here.

[1125] Object placement in virtual space

[1126] The device also performs the procedure to place the generated object in the virtual space as a 3D model. This allows users to intuitively place the objects they have created in the virtual space and share them with other users. For example, a generated "wooden bench" can be placed in the lobby of a virtual office.

[1127] Realization of in-house office

[1128] The server then uses the AI ​​model to generate non-player characters (NPCs). These NPCs have various roles, such as acting as meeting advisors or idea-boosting assistants. The generated NPC data is sent from the server to the device, which can then use it to place the NPCs in the virtual space.

[1129] User participation and interaction

[1130] Users participate in the virtual space as avatars. They can select their own avatar, log in, and interact with generated objects and NPCs. In particular, NPCs not only interact with users, but also with each other, providing new ideas and recreating the lively atmosphere of an office.

[1131] Applying the Emotion Engine

[1132] The system of the present invention further incorporates an emotion engine that recognizes the user's emotions. The emotion engine analyzes the user's voice data and facial expression data in real time to identify the user's emotional state. For example, if the user looks happy, the emotion engine will recognize this as "joy."

[1133] Based on the analysis results, the server adjusts the NPC's responses and behavior. For example, if the user looks dissatisfied, the NPC will respond by offering words of encouragement or suggesting solutions. The emotion engine's analysis results are also reflected in the dialogue between NPCs, resulting in more plausible interactions.

[1134] Specific examples

[1135] For example, a user selects "Object Creation" on the Metaverse management screen and enters the prompt "wooden bench." The server sends this prompt to the generation AI, which generates a Python script. The server then sends the script to the device, which then runs the script in Blender to generate a 3D model of the "wooden bench." The generated model is placed in the office lobby in the virtual space.

[1136] Meanwhile, the server uses an AI model to generate an NPC that acts as an "idea-boosting assistant" and sends that data to the device. The device places this NPC in the virtual space, and the user logs in as an avatar and interacts with the NPC, allowing the idea-boosting process to take place in the virtual space.

[1137] Furthermore, if the user expresses dissatisfaction during the interaction, the emotion engine will analyze and recognize this information in real time, and the server will adjust the NPC's response accordingly, providing specific advice and solutions to the user's problems. In this way, the emotion engine optimizes the interaction and improves the user experience.

[1138] As described above, the system of the present invention allows users to easily generate and place 3D objects, realizing a series of processes that recreate a real-world office experience in a virtual space. Furthermore, by combining it with an emotion engine, it is possible to optimize interactions and increase user satisfaction.

[1139] The processing flow will be explained below.

[1140] Object creation process flow

[1141] Step 1:

[1142] A user accesses the Metaverse administration screen and selects "Create Object." They are prompted to enter the desired object's characteristics and shape, or they can upload an existing image.

[1143] Step 2:

[1144] The server receives input data (prompts or images) from the user and sends it to the generation AI, which then automatically creates a Python script to generate an object based on the specified prompt or image.

[1145] Step 3:

[1146] The server sends the generated Python script to the user's device, which receives it and saves it to its local disk.

[1147] Step 4:

[1148] The terminal runs the saved Python script in 3D modeling software (e.g., Blender), which generates a 3D object based on the specified properties and shape.

[1149] Processing flow for placing objects in virtual space

[1150] Step 5:

[1151] The device performs a procedure for placing the generated 3D object in a virtual space, specifically, exporting or placing the object generated using the 3D modeling software at a specified location in the virtual space.

[1152] Process flow for realizing in-house office

[1153] Step 1:

[1154] The server uses the AI ​​model to generate an NPC with the role specified by the user (for example, an idea-boosting assistant). This NPC has the data necessary to interact with the user and communicate with other NPCs.

[1155] Step 2:

[1156] The server sends the generated NPC data to the user's device, which receives the data and places it in the virtual space.

[1157] Step 3:

[1158] Once the device has completed placing the NPC, the NPC becomes active in the virtual space, allowing for conversation and interaction with the user and other NPCs.

[1159] User participation and interaction process flow

[1160] Step 1:

[1161] A user logs into a virtual space as an avatar. The user selects their avatar and completes the login process.

[1162] Step 2:

[1163] The user can start interacting with the generated objects and NPCs. For example, they can provide ideas to NPCs placed as idea-boosting assistants and receive advice or new input in response.

[1164] Process flow for applying the emotion engine

[1165] Step 1:

[1166] When a user interacts in a virtual space, the emotion engine collects the user's voice data and facial expression data in real time, analyzes this data, and identifies the user's emotional state.

[1167] Step 2:

[1168] The server receives the analysis results of the emotion engine and adjusts the NPC's response. For example, if the user has a dissatisfied expression, the NPC will respond by offering words of encouragement or suggesting a solution.

[1169] Step 3:

[1170] The server adjusts the dialogue between other NPCs based on the results of the emotion engine, enabling more natural communication within the virtual space.

[1171] Specific examples

[1172] A concrete example of object creation

[1173] For example, a user selects "Object Creation" on the Metaverse management screen and enters the prompt "wooden bench." The server sends this prompt to the generation AI, which generates a Python script. The server then sends the script to the device, which then runs the script in Blender to generate a 3D model of the "wooden bench." The generated model is placed in the office lobby in the virtual space.

[1174] Example of NPC generation and dialogue

[1175] The server uses an AI model to generate an NPC that acts as an "idea-boosting assistant" and sends that data to the device. The device places this NPC in the virtual space, and the user logs in as an avatar and interacts with the NPC, allowing the idea-boosting process to take place in the virtual space.

[1176] Specific examples of emotion engine application

[1177] If the user expresses dissatisfaction during the interaction, the emotion engine will analyze and recognize this information in real time, and the server will adjust the NPC's response accordingly, providing specific advice and solutions to the user's problems. In this way, the emotion engine optimizes the interaction and improves the user experience.

[1178] As described above, the system of the present invention allows users to easily generate and place 3D objects, realizing a series of processes that recreate a real-world office experience in a virtual space. Furthermore, by combining it with an emotion engine, the user experience can be further improved.

[1179] Example 2

[1180] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1181] Interaction in modern virtual environments requires a method that allows users to easily create and place objects. However, existing systems require specialized knowledge and numerous manual operations, and few are easy to use. Furthermore, there is a lack of means to accurately reflect the user's emotions and achieve more natural and effective interactions in virtual environments, limiting the quality of the user experience.

[1182] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1183] In this invention, the server includes: a prompt or image input means for a user to specify a desired object; a generation AI generating code based on the specified prompt or image; a 3D model creation software creating an object using the generated code; a virtual environment placing the created object; and a sentiment analysis engine recognizing the user's sentiment and adjusting the responses and behavior of a virtual character based on the recognition results. This allows users to easily create and place 3D objects without requiring specialized knowledge, and enables natural and effective dialogue within the virtual environment utilizing sentiment analysis.

[1184] A "prompt" is input data that describes the target or condition desired by the user in natural language.

[1185] An "image" is digital data that visually represents an object or condition specified by a user.

[1186] "Generative AI" is an artificial intelligence system that analyzes user input data and automatically generates the necessary code and models.

[1187] "Code" refers to the program instructions generated by the AI ​​based on prompts and images specified by the user.

[1188] "3D model creation software" is software that actually generates 3D objects based on generated code.

[1189] A "virtual environment" is a computer-generated simulated space in which a user can interact.

[1190] A "virtual character" is an NPC (non-player character) in a virtual environment, a character that is set up to converse and interact with the user.

[1191] An "emotion analysis engine" is a technology that analyzes a user's voice, facial expressions, etc. to identify their emotional state.

[1192] "User's computer" means an information processing device used by a user to receive data from the generated AI and use it within the virtual environment.

[1193] An "information processing device" is a device that transmits, receives, analyzes, and stores digital data, and examples of this include servers.

[1194] The present invention provides a system that allows a user to easily create desired objects and place them in a virtual environment. Specific embodiments of the system will be described below.

[1195] Hardware and software used

[1196] This system mainly uses the following hardware and software:

[1197] Server: Receives prompts and image data and sends them to the generation AI

[1198] Terminal: A computer operated by a user, with 3D modeling software such as Blender installed.

[1199] Generative AI model: Analyzes user input data and generates the necessary Python scripts

[1200] Virtual environment management software: Place and manage objects and virtual characters in a virtual environment

[1201] Emotion Analysis Engine: Analyzes user emotions and adjusts the virtual character's responses and behavior

[1202] Detailed System Description

[1203] 1. Object creation prompt input

[1204] A user opens a browser and logs into the Metaverse admin page, where they select the "Create Object" option and enter the desired object's characteristics and shape as a prompt, or upload an existing image (e.g., "Wooden Bench").

[1205] Example prompt sentence:

[1206] "Generate a wooden bench."

[1207] "Create a virtual character that acts as a meeting advisor."

[1208] 2. Generate prompts and send them to the AI

[1209] The server receives user input and provides prompts and image data to the generative AI, which then creates a Python script to generate the object. This script contains commands and parameters that can be executed in 3D modeling software such as Blender.

[1210] 3. Generate and run the Python script

[1211] The AI ​​analyzes the prompt and automatically generates a Python script, which is then sent to the user's device via a server. The device then uses 3D modeling software such as Blender to generate objects based on the script.

[1212] 4. Placing objects in the virtual environment

[1213] The device places the generated 3D model in the virtual environment, and the user can adjust the object's exact position using the mouse or keyboard. Information about the placed object is registered in the virtual environment management software and shared with other users.

[1214] 5. Virtual Character Generation and Emotional Response

[1215] The server uses AI models to generate virtual characters, which can act as meeting advisors, idea-boosting assistants, etc. These characters are sent to the device based on the data generated by the AI ​​and placed in the virtual environment.

[1216] In addition, an emotion analysis engine analyzes the user's voice and facial expressions in real time and adjusts the virtual character's responses and behavior accordingly. For example, if the user looks unhappy, the virtual character will offer words of encouragement or suggest solutions to the problem.

[1217] Specific examples

[1218] For example, consider the case of generating a "wooden bench." The user selects "Object Creation" on the metaverse management screen and enters "wooden bench" as a prompt. The server sends this prompt to the generation AI, which creates a Python script. The server then sends the script to the device, which then runs it in Blender. The resulting 3D model of the "wooden bench" is placed in the lobby of the virtual office.

[1219] In addition, a virtual character called a "meeting advisor" is generated, and when a user interacts with the virtual environment, an emotion analysis engine analyzes the user's emotional state in real time and adjusts the virtual character's responses accordingly, allowing users to experience more natural and effective interactions within the virtual environment.

[1220] The above is a concrete example of how to implement the invention. This system allows users to easily generate 3D objects without requiring specialized knowledge, and enables effective dialogue utilizing emotion analysis.

[1221] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1222] Step 1:

[1223] The user opens a browser and accesses the metaverse management screen. The login screen is displayed, the user enters their username and password, and clicks the login button. The entered username and password are sent to the server for authentication. If authentication is successful, the metaverse management screen is displayed.

[1224] Input: Username, Password

[1225] Output: Metaverse admin page

[1226] Step 2:

[1227] On the Metaverse Admin screen, select the "Create Object" option. The user enters the desired object's characteristics and shape as a prompt, or uploads an existing image. For example, enter the prompt "Wooden bench." The user's input is sent to the server.

[1228] Input: A prompt or image (e.g., "Wooden bench")

[1229] Output: Prompt data sent to the server

[1230] Step 3:

[1231] The server takes the user's input and sends it to a generative AI, which parses the prompt and generates a Python script containing commands and parameters that can be executed in 3D modeling software such as Blender.

[1232] Input: prompt data

[1233] Output: Python script

[1234] Step 4:

[1235] The generated Python script is sent from the server to the device, where it is executed in 3D modeling software such as Blender. The specified object is generated based on the script.

[1236] Input: Python script

[1237] Output: 3D object

[1238] Step 5:

[1239] The device places the generated 3D object in the virtual environment. The user can adjust the object's position using the mouse or keyboard. The information about the placed object is saved in the virtual environment management software and can be shared with other users.

[1240] Input: 3D object

[1241] Output: Objects placed in the virtual environment

[1242] Step 6:

[1243] The server uses an AI model to generate a virtual character, which can act as a meeting advisor or idea-boosting assistant, for example. The generated character data is sent from the server to the device, which then places it in the virtual environment.

[1244] Input: Virtual character role (e.g., conference advisor)

[1245] Output: Generated virtual character data

[1246] Step 7:

[1247] The device places a virtual character in the virtual environment. When a user logs into the virtual environment as an avatar, they can begin interacting with the virtual character. An emotion analysis engine analyzes the user's emotions in real time and adjusts the virtual character's responses and behavior based on the results.

[1248] Input: Virtual character data, user emotion data

[1249] Output: User interaction with virtual character

[1250] Through the above process, this system allows users to easily create and place desired objects and effectively interact with them in the virtual environment.

[1251] (Application example 2)

[1252] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1253] Unlike real stores, creating product displays and layouts in virtual stores can be non-intuitive and require specialized skills, so there is a demand for technology that allows non-experts to easily create and arrange displays. Furthermore, there is a lack of means to optimize interactions by reflecting human emotions, which can degrade the quality of the user experience. Therefore, there is a demand for a system that can easily and effectively generate objects in virtual spaces and optimize interactions through emotion recognition.

[1254] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for a user to specify a desired object using a prompt or an image; means for a generation AI to generate a Python script based on the specified prompt or image; means for creating an object using 3D modeling software using the generated Python script; means for placing the created object in a virtual space; emotion recognition means for analyzing the user's emotions in real time; and means for adjusting the responses of NPCs based on the emotion recognition results. This allows even non-experts to intuitively generate and place objects in a virtual space, and further optimizes interactions according to the user's emotions, enabling a high-quality user experience.

[1255] A "prompt" is text or a command that a user enters into a computer system to instruct or request something.

[1256] An "image" is digital data representing visual information that can be displayed and processed on a computer.

[1257] "Generative AI" refers to artificial intelligence technology that automatically generates new data (in this case, Python scripts) based on input data.

[1258] A "Python script" is a series of instructions or procedures written in the Python programming language.

[1259] "3D modeling software" refers to software for creating and editing three-dimensional objects and scenes on a computer.

[1260] A "virtual space" is a computer-generated digital environment or simulation.

[1261] "Emotion recognition means" refers to technology or systems that analyze a user's facial expressions and voice data to identify their emotional state.

[1262] "NPC" stands for "Non-Player-Character" and refers to a virtual character that is not controlled by the user but can be interacted with.

[1263] MODE FOR CARRYING OUT THE INVENTION

[1264] The following describes an embodiment of the present invention. The system of the present invention allows a user to specify a desired object using a prompt or an image, automatically generates a Python script using generative AI, creates the object using 3D modeling software, and places the object in a virtual space. Furthermore, an emotion engine that recognizes the user's emotions is used to optimize interactions within the virtual space.

[1265] Hardware and software used

[1266] The system is implemented using the following hardware and software:

[1267] Server: Hosts the generative AI model and emotion recognition engine, and processes prompts and image data.

[1268] User's device: This includes smart glasses, smartphones, head-mounted displays, etc., used by users to access the virtual space.

[1269] Generative AI model: Generates Python scripts based on user prompts and images.

[1270] 3D modeling software: Execute the script generated using Blender or similar software to generate a 3D model.

[1271] Emotion recognition engine: Analyzes the user's facial expressions and voice data in real time to provide appropriate interactions.

[1272] Data processing and calculation

[1273] The server receives prompts and image data entered by the user and sends them to the generative AI model, which then generates a Python script based on the input and sends it back to the server. The server then sends the Python script to the user's device, which then runs the script in the 3D modeling software.

[1274] The generated 3D model is placed in the virtual space on the user's device. The device is also equipped with an emotion recognition engine that analyzes the user's emotional data in real time. Based on this emotional data, the server adjusts the responses and behavior of the NPC (virtual character) to provide a satisfying interaction for the user.

[1275] Specific examples

[1276] For example, a user puts on smart glasses and accesses a virtual store management screen, selects an empty display shelf in the store, and enters the prompt to "create a new display" or uploads an image of an existing display.

[1277] The server receives user input and sends prompts and image data to the generative AI model, which generates a suitable Python script in response to the prompt "Create a new display." This script is then sent from the server to the user's smart glasses, where it is generated and placed as a 3D object on a virtual display within the glasses.

[1278] The user checks the generated display in the virtual store and fine-tunes its placement as needed. The emotion recognition engine reads the user's facial expressions and voice to analyze their emotional state. For example, if the user shows a satisfied expression, the emotion engine recognizes this and notifies the user that the generation process was successful.

[1279] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1280] Step 1:

[1281] Users access the virtual store management screen using devices such as smart glasses or smartphones, where they can enter a prompt to create a desired object (e.g., "Create a new display") or upload an image of an existing display. The prompt or image data is provided as input.

[1282] Step 2:

[1283] The server receives prompts and image data entered by the user and sends them to the generative AI model. Specifically, the prompts and image data are used as input, and the generative AI model analyzes them. Here, preprocessing and tokenization of the data are performed.

[1284] Step 3:

[1285] The generative AI model generates a Python script based on the received prompt and image data. The generated Python script is output. The generative AI model automatically creates a script based on the specified prompt.

[1286] Step 4:

[1287] The server sends the generated Python script to the user's terminal. This is the process by which the generated script is transferred from the server to the user's terminal. The output from the server is a Python script, and the input to the terminal is the same script.

[1288] Step 5:

[1289] The user's device executes the received Python script in 3D modeling software (such as Blender) to generate a 3D object. In this step, the script is actually executed and a 3D model is generated. The input is the Python script, and the output is the 3D model.

[1290] Step 6:

[1291] The user's device places the generated 3D model in a virtual space. Specifically, the 3D model is placed in a virtual store as a virtual display. The input is the 3D model, and the output is placement in the virtual space.

[1292] Step 7:

[1293] The emotion recognition engine installed in the device analyzes the user's facial expressions and voice in real time to recognize their emotional state. The input is the user's facial expression data and voice data, and the emotional state is recognized as data calculation. Emotional data based on the analysis is output.

[1294] Step 8:

[1295] The server adjusts the NPC's responses and behavior based on the emotional data obtained from the emotion recognition engine. The input is the output data of the emotion recognition engine, and the server processes the data to adjust the NPC's responses and behavior. The adjusted NPC settings are obtained as the output.

[1296] Step 9:

[1297] The user interacts with the generated displays and NPCs in the virtual space and fine-tunes their placement as needed. This step optimizes the user experience. Specifically, the user can view the generated displays and receive feedback based on emotion recognition. The input is the user's behavior and emotion data, and the output is an optimized user experience.

[1298] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1299] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1300] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1301] [Fourth embodiment]

[1302] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1303] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1304] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1305] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1306] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1307] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1308] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1309] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1310] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1311] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1312] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1313] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1314] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1315] The system of the present invention allows the user to specify the desired object using prompts or images, automatically generates a Python script using generation AI, creates the object using 3D modeling software, and places the object in virtual space.

[1316] Object Creation

[1317] First, a user accesses the Metaverse administration screen and selects the object creation option. The user enters a prompt for the desired object's characteristics and shape, or uploads an existing image. For example, a user might enter the prompt "wooden bench."

[1318] The server then receives the user's input and provides prompts and image data to the generation AI, which then creates a Python script to generate the object. This Python script, when executed in 3D modeling software, contains the specific steps and parameters required to generate the specified object.

[1319] The Python script generated by the generative AI is sent from the server to the terminal, and the terminal generates an object using 3D modeling software based on this script. Common 3D modeling software such as Blender can be used here.

[1320] Object placement in virtual space

[1321] The device also performs the procedure to place the generated object in the virtual space as a 3D model. This allows users to intuitively place the objects they have created in the virtual space and share them with other users. For example, a generated "wooden bench" can be placed in the lobby of a virtual office.

[1322] Realization of in-house office

[1323] The server then uses the AI ​​model to generate non-player characters (NPCs). These NPCs have various roles, such as acting as advisors in meetings or as idea-boosting assistants. The generated NPC data is sent from the server to the device, which can then use it to place the NPCs in the virtual space.

[1324] User participation and interaction

[1325] Finally, users participate in the virtual space as avatars. Users can select their own avatars, log in, and interact with generated objects and NPCs. In particular, NPCs not only interact with users, but also with each other, providing new ideas and recreating the lively atmosphere of an office.

[1326] Specific examples

[1327] For example, a user selects "Object Creation" on the Metaverse management screen and enters the prompt "wooden bench." The server sends this prompt to the generation AI, which generates a Python script. The server then sends the script to the device, which then runs the script in Blender to generate a 3D model of the "wooden bench." The generated model is placed in the office lobby in the virtual space.

[1328] Meanwhile, the server uses an AI model to generate an NPC that acts as an "idea-boosting assistant" and sends that data to the device. The device places this NPC in the virtual space, and the user logs in as an avatar and interacts with the NPC, allowing the idea-boosting process to take place in the virtual space.

[1329] As described above, the system of the present invention enables users to easily generate and place 3D objects and recreate the experience of a real-world office in a virtual space.

[1330] The processing flow will be explained below.

[1331] Object creation process flow

[1332] Step 1:

[1333] A user accesses the Metaverse administration screen and selects "Create Object," where they are prompted to enter the desired object's characteristics and shape, or upload an existing image.

[1334] Step 2:

[1335] The server receives input data (prompts or images) from the user and sends it to the generation AI, which then automatically creates a Python script to generate an object based on the specified prompt or image.

[1336] Step 3:

[1337] The server sends the generated Python script to the user's device, which receives it and saves it to its local disk.

[1338] Step 4:

[1339] The terminal runs the saved Python script in 3D modeling software (e.g., Blender), which generates a 3D object based on the specified properties and shape.

[1340] Object placement in virtual space

[1341] Step 5:

[1342] The device performs a procedure for placing the generated 3D object in a virtual space, specifically, exporting or placing the object generated using the 3D modeling software at a specified location in the virtual space.

[1343] Realization of in-house office

[1344] Step 1:

[1345] The server uses the AI ​​model to generate an NPC with the role specified by the user (for example, an idea-boosting assistant). This NPC has the data necessary to interact with the user and communicate with other NPCs.

[1346] Step 2:

[1347] The server sends the generated NPC data to the user's device, which receives the data and places it in the virtual space.

[1348] Step 3:

[1349] Once the device has completed placing the NPC, the NPC becomes active in the virtual space, allowing for conversation and interaction with the user and other NPCs.

[1350] User participation and interaction

[1351] Step 1:

[1352] A user logs into a virtual space as an avatar. The user selects their avatar and completes the login process.

[1353] Step 2:

[1354] The user can start interacting with the generated objects and NPCs. For example, they can provide ideas to NPCs placed as idea-boosting assistants and receive advice or new input in response.

[1355] As described above, the system of the present invention realizes a series of processes that allow users to easily generate and place 3D objects and recreate a real-world office experience in a virtual space.

[1356] Example 1

[1357] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1358] Conventional systems for generating 3D objects and placing them in virtual spaces are difficult to use unless the user has a high level of expertise. Furthermore, the interaction functions with generated objects and NPCs (Non-Player-Characters) are insufficient, resulting in poor user experience and reduced usability. Furthermore, security and processing speed issues frequently arise when exchanging generated data.

[1359] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1360] In this invention, the server includes an input means for a user to input a prompt or an image, a means for a generative AI model to generate a Python script based on the input prompt or image, and a means for creating an object with 3D modeling software using the generated Python script, which enables even users without specialized knowledge to easily generate 3D objects and intuitively place them in a virtual space.

[1361] In this invention, the server further includes a means for allowing users to participate in the virtual space as avatars, a means for generating and placing NPCs (Non-Player-Characters) in the virtual space and for the NPCs to interact with the user, and a means for the user to refer to information generated by the interactions between NPCs, thereby enabling richer interactions in the virtual space and improving the user experience.

[1362] This invention also includes a means for using a combination of an image generation algorithm and a 3D modeling algorithm to generate an object using a generative AI model, a means for transmitting the generated 3D model to a user's terminal via a server, and a means for placing the transmitted 3D model in a virtual space on the user's terminal, thereby enabling the generated data to be exchanged efficiently and safely and quickly placed in the virtual space.

[1363] "prompt"

[1364] is an input format in which a user specifies the desired properties and shape of an object in text format.

[1365] "image"

[1366] is visual data that a user uploads to visually specify an existing object.

[1367] "Generative AI model"

[1368] is a system that includes Python scripts and artificial intelligence algorithms for generating 3D objects based on user input data.

[1369] "Python script"

[1370] is program code that is automatically generated by a generative AI model and contains instructions and parameters for creating an object in 3D modeling software.

[1371] "3D modeling software"

[1372] is a software application that runs Python scripts to create and edit 3D objects. Examples include Blender.

[1373] "Object"

[1374] is a digital model with three-dimensional shape and structure that is generated to be placed in a virtual space.

[1375] "Virtual Space"

[1376] is a computer-generated environment built for users to interact with digital objects and NPCs.

[1377] "Users"

[1378] A player is a person who uses the system to input prompts or images to interact with generated objects and NPCs.

[1379] "Input means"

[1380] is the interface through which the user provides prompts and images to the system.

[1381] "Generation means"

[1382] is a component that performs processing to generate an object based on the input prompt and image.

[1383] "Method of Creation"

[1384] is a method for running the generated Python script to create the object in 3D modeling software.

[1385] "Arrangement means"

[1386] is a tool or process for properly positioning generated objects in a virtual space.

[1387] "Avatar"

[1388] is a digital character that users use to represent themselves in a virtual space.

[1389] "NPC (Non-Player-Character)"

[1390] is a digital character generated by the system that interacts with the user in a virtual space.

[1391] "Image generation algorithm"

[1392] is an algorithm that manages the process of generating objects based on user-provided image data.

[1393] "3D modeling algorithm"

[1394] is an algorithm for calculating and generating 3D shapes used by generative AI models.

[1395] "Transmission means"

[1396] is a protocol and function for transferring generated 3D models and data from a server to a user's device.

[1397] The system of the present invention allows the user to specify the desired object using prompts or images, automatically generates a Python script using a generative AI model, creates the object using 3D modeling software, and places the object in a virtual space.

[1398] Hardware and Software Configuration

[1399] Server: A high-performance server machine is used to run the generative AI model and process the data.

[1400] Terminal: Use a personal computer to accept user input, run generated Python scripts, and place 3D objects in the virtual space.

[1401] Generative AI model: An artificial intelligence model for generating Python scripts based on user prompts and images. For example, we use a natural language processing model (e.g., ChatGPT).

[1402] 3D modeling software: Software that runs Python scripts to create and edit 3D objects. Blender will be used as an example.

[1403] Object creation procedure

[1404] 1. A user accesses the Metaverse Admin screen and selects the object creation option. Here, the user enters the desired object characteristics and shape as a prompt, or uploads an existing image. In this example, the user enters the prompt "wooden bench."

[1405] 2. The server receives the user's input data (prompts and images) and provides the prompts and image data to the generative AI model. The generative AI model then creates a Python script to generate the object. This Python script contains the specific steps and parameters to be executed in the 3D modeling software.

[1406] 3. The generated Python script is sent from the server to the terminal, which then launches 3D modeling software such as Blender and executes the Python script to generate the specified object.

[1407] Object placement in virtual space

[1408] The device then performs the procedure to place the generated 3D object in the virtual space. Specifically, the user can intuitively drag and drop the object within the virtual space to place it. For example, placing a generated "wooden bench" in the lobby of a virtual office corresponds to this operation.

[1409] Realization of in-house office

[1410] The server uses AI models to generate non-player characters (NPCs). These NPCs have various roles, such as acting as advisors in meetings or as idea-boosting assistants. The generated NPC data is then sent from the server to the device, which can then use it to place the NPCs in the virtual space.

[1411] User participation and interaction

[1412] Users participate in the virtual space as avatars. They can select their own avatar, log in, and interact with generated objects and NPCs. In particular, NPCs not only interact with users, but also with each other, providing new ideas and recreating the lively atmosphere of an office.

[1413] Specific examples

[1414] As a specific example, a case will be described in which a user selects "Create Object" on the metaverse management screen and inputs the prompt "Wooden Bench."

[1415] 1. The server sends this prompt to the generation AI, which generates a Python script.

[1416] 2. The generated script is sent from the server to the device, and the device executes the script in Blender to generate a 3D model of a "wooden bench."

[1417] 3. The user drags and drops the model into the virtual office lobby.

[1418] As another specific example, we will also explain a case where the server uses an AI model to generate an NPC with the role of an "idea bouncing assistant."

[1419] 1. The generated NPC data is sent from the server to the device.

[1420] 2. The device places this NPC in the virtual space, and the user logs in as an avatar and interacts with the NPC, bouncing ideas off each other in the virtual space.

[1421] As described above, the system of the present invention allows even users without specialized knowledge to easily generate 3D objects and intuitively place them in a virtual space.

[1422] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1423] Step 1:

[1424] The user accesses the metaverse management screen and selects the option for creating the object they desire.

[1425] Input: User access and object creation choices

[1426] Output: Object creation option selection

[1427] Step 2:

[1428] The user enters the desired object characteristics or shape as a prompt, or uploads an existing image. For example, enter the prompt "wooden bench."

[1429] Input: prompt or image data

[1430] Output: Prompt or image data is reflected in the Metaverse management screen

[1431] Step 3:

[1432] The server receives the user's input data (prompts and images).

[1433] Input: prompt or image data

[1434] Output: State that receives prompt or image data

[1435] Step 4:

[1436] The server provides prompts and image data to the generated AI model.

[1437] Input: prompt or image data

[1438] Output: Providing data to a generative AI model

[1439] Step 5:

[1440] Create a Python script that will allow the generative AI model to generate objects based on the provided prompts and image data, with specific steps and parameters to be executed in the 3D modeling software.

[1441] Input: prompt or image data

[1442] Output: Python script

[1443] Step 6:

[1444] The server sends the generated Python script to the device.

[1445] Input: Python script

[1446] Output: Script sending status to the terminal

[1447] Step 7:

[1448] Based on the Python script received by the device, 3D modeling software such as Blender is launched.

[1449] Input: Python script

[1450] Output: 3D modeling software running

[1451] Step 8:

[1452] The terminal will run the Python script in Blender and create the specified object.

[1453] Input: Python script

[1454] Output: Generated 3D object

[1455] Step 9:

[1456] The device provides an interface for placing the generated 3D objects in the virtual space. Users can intuitively drag and drop objects to place them in the virtual space. For example, placing a wooden bench in the lobby of a virtual office.

[1457] Input: Generated 3D object

[1458] Output: Objects placed in virtual space

[1459] Step 10:

[1460] The server generates NPCs (Non-Player-Characters) using an AI model and sends the data to the device.

[1461] Input: None (internal data)

[1462] Output: Send NPC data to the terminal

[1463] Step 11:

[1464] The device places NPCs generated by an AI model into the virtual space.

[1465] Input: NPC data

[1466] Output: NPC placed in virtual space

[1467] Step 12:

[1468] Users participate in the virtual space as avatars. Users select their own avatar and log in.

[1469] Input: Avatar selection and login information

[1470] Output: Avatar participation status in virtual space

[1471] Step 13:

[1472] Users interact with generated objects and NPCs through conversations. By observing the conversations and information sharing between NPCs, new ideas can be gained.

[1473] Input: User actions and interactions

[1474] Output: Dialogue results and new ideas

[1475] (Application example 1)

[1476] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1477] In modern virtual spaces and virtual stores, it is difficult for users to intuitively customize desired products and objects and then view and operate the generated objects. In particular, there is a demand for a system that allows users to easily generate specific objects without specialized knowledge and view and operate them in a virtual space. Another challenge is the placement of automatic characters to enhance user interaction and provide a more realistic experience, as well as the utilization of information generated by these characters.

[1478] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1479] In this invention, the server includes a means for a user to specify a desired object using a prompt or image, a means for a generation AI to generate a program based on the specified prompt or image, and a means for creating an object using 3D modeling software using the generated program. This allows users to easily generate desired objects without any special knowledge, place them in a virtual space, and view and manipulate them. Furthermore, to enhance the user's experience, automated characters can be placed and information generated through interactions between the user and the characters can be provided.

[1480] A "prompt" is a text input that allows a user to specify the desired properties or shape of an object.

[1481] An "image" is visual data uploaded by a user to specify the properties and shape of a desired object.

[1482] A "program" is a script automatically generated by generative AI to create an object in 3D modeling software.

[1483] "3D modeling software" refers to software tools for programmatically generating 3D objects.

[1484] A "virtual space" is a computer-generated space that users can interact with.

[1485] A "user" is an entity that uses the system to specify and create a desired object.

[1486] A "terminal" is a digital device used by a user, such as a computer or smartphone.

[1487] "Automatic characters" refer to non-player characters (NPCs) that can interact with users in a virtual space and with each other.

[1488] "Generative AI" is an artificial intelligence technology that automatically creates programs to generate objects based on input prompts and images.

[1489] An "algorithm" is a step-by-step procedure or formula for solving a particular problem.

[1490] The "information processing device" is a network-compatible device for transmitting the generated three-dimensional model to the user's terminal.

[1491] A "model" is generated three-dimensional object data, and refers to a specific design that is placed in a virtual space.

[1492] An "object" refers to any object or item designated by the user, and is a target object generated in a virtual space.

[1493] The embodiment of the present invention is based on a system comprising the following steps.

[1494] First, the user accesses the virtual environment management screen using a smartphone or personal computer. This management screen provides multiple options for creating the object the user desires. The user specifies the desired object by entering a prompt or uploading an image. Examples of prompts include a "wooden bench" or a "stylish glass table."

[1495] The user's input data is then sent to a server, which uses generative AI to automatically generate a program to generate a concrete object from the prompt or image. This program is written as a Python script and can be executed in 3D modeling software (e.g., Blender).

[1496] The generated Python script is sent to the user's terminal via the network, where it is executed by the 3D modeling software to generate a specific object.

[1497] The generated 3D model is then placed directly in a virtual space on the device, allowing the user to visually confirm the object of their choice within the virtual space. This virtual space can be a virtual store, and the user can place the generated object in real life for use in sales promotions, interior design simulations, and more.

[1498] Furthermore, the server can generate automated characters and place them in the virtual space. This allows users to interact with the automated characters, resulting in a more realistic experience. The automated characters can also interact with each other, and users can view the content of their conversations and the information generated.

[1499] Hardware and software used

[1500] Server: A computer server for running generative AI

[1501] Device: A smartphone or personal computer operated by a user

[1502] 3D modeling software: Blender, etc.

[1503] Generative AI: Artificial intelligence technology that generates programs based on prompts and images

[1504] Specific examples

[1505] 1. The user selects "Object Creation" on the virtual space management screen and enters the prompt "Wooden bench."

[1506] 2. The server sends this prompt to the generation AI, which generates a Python script.

[1507] 3. The generated script is sent from the server to the device, and the device executes the script in Blender to generate a three-dimensional model of the "wooden bench."

[1508] 4. The generated model is placed in the virtual store showroom.

[1509] In this way, the present invention enables users to easily create and place desired objects and freely manipulate them in virtual space without any special knowledge.

[1510] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1511] Step 1:

[1512] A user accesses the virtual space management screen and selects the "Object Creation" option. The user is prompted to enter the characteristics and shape of an object, such as a "wooden bench," or upload an existing image.

[1513] Input: User input (prompt or image)

[1514] Output: Input prompt and image data

[1515] Step 2:

[1516] The server receives user input and sends the data to the Generative AI, which generates a Python script to generate the object based on the prompt and image.

[1517] Input: User input (prompt or image)

[1518] Data processing: Calculation process to generate Python scripts based on prompt statements

[1519] Output: Generated Python script

[1520] Step 3:

[1521] The generated Python script is sent from the server to the user's terminal.

[1522] Input: Generated Python script

[1523] Output: Python script sent to user's terminal

[1524] Step 4:

[1525] The terminal executes the received Python script in 3D modeling software (e.g., Blender) to generate an object.

[1526] Input: The received Python script

[1527] Data processing: 3D model generation processing using 3D modeling software

[1528] Output: Generated 3D model

[1529] Step 5:

[1530] The generated three-dimensional model is placed in a virtual space on the terminal, and the user can visually check and operate it.

[1531] Input: Generated 3D model

[1532] Output: 3D model placed in the virtual space on the device

[1533] Step 6:

[1534] The server uses a generation AI to generate automated characters and place them in the virtual space, allowing users to interact with them.

[1535] Input: Automatic character generation request

[1536] Data processing: AI calculation processing for automatic character generation

[1537] Output: Generated auto character

[1538] Step 7:

[1539] The automated characters converse with the user and with each other, and the content of the conversation can be viewed by the user.

[1540] Input: User interaction with automated character

[1541] Output: Interactions and generated information

[1542] Step 8:

[1543] The generated information is sent to the user's terminal, where the user can check the details.

[1544] Input: Generated information

[1545] Output: Display information on the user's terminal

[1546] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1547] The system of the present invention allows users to specify the desired object using prompts or images, automatically generates a Python script using a generation AI, creates the object using 3D modeling software, and places the object in a virtual space. Furthermore, it combines an emotion engine that recognizes the user's emotions to optimize interactions within the virtual space.

[1548] Object Creation

[1549] First, a user accesses the Metaverse administration screen and selects the object creation option. The user enters a prompt for the desired object's characteristics and shape, or uploads an existing image. For example, a user might enter the prompt "wooden bench."

[1550] The server then receives the user's input and provides prompts and image data to the generation AI, which then creates a Python script to generate the object. This Python script, when executed in 3D modeling software, contains the specific steps and parameters required to generate the specified object.

[1551] The Python script generated by the generative AI is sent from the server to the terminal, and the terminal generates an object using 3D modeling software based on this script. Common 3D modeling software such as Blender can be used here.

[1552] Object placement in virtual space

[1553] The device also performs the procedure to place the generated object in the virtual space as a 3D model. This allows users to intuitively place the objects they have created in the virtual space and share them with other users. For example, a generated "wooden bench" can be placed in the lobby of a virtual office.

[1554] Realization of in-house office

[1555] The server then uses the AI ​​model to generate non-player characters (NPCs). These NPCs have various roles, such as acting as meeting advisors or idea-boosting assistants. The generated NPC data is sent from the server to the device, which can then use it to place the NPCs in the virtual space.

[1556] User participation and interaction

[1557] Users participate in the virtual space as avatars. They can select their own avatar, log in, and interact with generated objects and NPCs. In particular, NPCs not only interact with users, but also with each other, providing new ideas and recreating the lively atmosphere of an office.

[1558] Applying the Emotion Engine

[1559] The system of the present invention further incorporates an emotion engine that recognizes the user's emotions. The emotion engine analyzes the user's voice data and facial expression data in real time to identify the user's emotional state. For example, if the user looks happy, the emotion engine will recognize this as "joy."

[1560] Based on the analysis results, the server adjusts the NPC's responses and behavior. For example, if the user looks dissatisfied, the NPC will respond by offering words of encouragement or suggesting solutions. The emotion engine's analysis results are also reflected in the dialogue between NPCs, resulting in more plausible interactions.

[1561] Specific examples

[1562] For example, a user selects "Object Creation" on the Metaverse management screen and enters the prompt "wooden bench." The server sends this prompt to the generation AI, which generates a Python script. The server then sends the script to the device, which then runs the script in Blender to generate a 3D model of the "wooden bench." The generated model is placed in the office lobby in the virtual space.

[1563] Meanwhile, the server uses an AI model to generate an NPC that acts as an "idea-boosting assistant" and sends that data to the device. The device places this NPC in the virtual space, and the user logs in as an avatar and interacts with the NPC, allowing the idea-boosting process to take place in the virtual space.

[1564] Furthermore, if the user expresses dissatisfaction during the interaction, the emotion engine will analyze and recognize this information in real time, and the server will adjust the NPC's response accordingly, providing specific advice and solutions to the user's problems. In this way, the emotion engine optimizes the interaction and improves the user experience.

[1565] As described above, the system of the present invention allows users to easily generate and place 3D objects, realizing a series of processes that recreate a real-world office experience in a virtual space. Furthermore, by combining it with an emotion engine, it is possible to optimize interactions and increase user satisfaction.

[1566] The processing flow will be explained below.

[1567] Object creation process flow

[1568] Step 1:

[1569] A user accesses the Metaverse administration screen and selects "Create Object." They are prompted to enter the desired object's characteristics and shape, or they can upload an existing image.

[1570] Step 2:

[1571] The server receives input data (prompts or images) from the user and sends it to the generation AI, which then automatically creates a Python script to generate an object based on the specified prompt or image.

[1572] Step 3:

[1573] The server sends the generated Python script to the user's device, which receives it and saves it to its local disk.

[1574] Step 4:

[1575] The terminal runs the saved Python script in 3D modeling software (e.g., Blender), which generates a 3D object based on the specified properties and shape.

[1576] Processing flow for placing objects in virtual space

[1577] Step 5:

[1578] The device performs a procedure for placing the generated 3D object in a virtual space, specifically, exporting or placing the object generated using the 3D modeling software at a specified location in the virtual space.

[1579] Process flow for realizing in-house office

[1580] Step 1:

[1581] The server uses the AI ​​model to generate an NPC with the role specified by the user (for example, an idea-boosting assistant). This NPC has the data necessary to interact with the user and communicate with other NPCs.

[1582] Step 2:

[1583] The server sends the generated NPC data to the user's device, which receives the data and places it in the virtual space.

[1584] Step 3:

[1585] Once the device has completed placing the NPC, the NPC becomes active in the virtual space, allowing for conversation and interaction with the user and other NPCs.

[1586] User participation and interaction process flow

[1587] Step 1:

[1588] A user logs into a virtual space as an avatar. The user selects their avatar and completes the login process.

[1589] Step 2:

[1590] The user can start interacting with the generated objects and NPCs. For example, they can provide ideas to NPCs placed as idea-boosting assistants and receive advice or new input in response.

[1591] Process flow for applying the emotion engine

[1592] Step 1:

[1593] When a user interacts in a virtual space, the emotion engine collects the user's voice data and facial expression data in real time, analyzes this data, and identifies the user's emotional state.

[1594] Step 2:

[1595] The server receives the analysis results of the emotion engine and adjusts the NPC's response. For example, if the user has a dissatisfied expression, the NPC will respond by offering words of encouragement or suggesting a solution.

[1596] Step 3:

[1597] The server adjusts the dialogue between other NPCs based on the results of the emotion engine, enabling more natural communication within the virtual space.

[1598] Specific examples

[1599] A concrete example of object creation

[1600] For example, a user selects "Object Creation" on the Metaverse management screen and enters the prompt "wooden bench." The server sends this prompt to the generation AI, which generates a Python script. The server then sends the script to the device, which then runs the script in Blender to generate a 3D model of the "wooden bench." The generated model is placed in the office lobby in the virtual space.

[1601] Example of NPC generation and dialogue

[1602] The server uses an AI model to generate an NPC that acts as an "idea-boosting assistant" and sends that data to the device. The device places this NPC in the virtual space, and the user logs in as an avatar and interacts with the NPC, allowing the idea-boosting process to take place in the virtual space.

[1603] Specific examples of emotion engine application

[1604] If the user expresses dissatisfaction during the interaction, the emotion engine will analyze and recognize this information in real time, and the server will adjust the NPC's response accordingly, providing specific advice and solutions to the user's problems. In this way, the emotion engine optimizes the interaction and improves the user experience.

[1605] As described above, the system of the present invention allows users to easily generate and place 3D objects, realizing a series of processes that recreate a real-world office experience in a virtual space. Furthermore, by combining it with an emotion engine, the user experience can be further improved.

[1606] Example 2

[1607] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1608] Interaction in modern virtual environments requires a method that allows users to easily create and place objects. However, existing systems require specialized knowledge and numerous manual operations, and few are easy to use. Furthermore, there is a lack of means to accurately reflect the user's emotions and achieve more natural and effective interactions in virtual environments, limiting the quality of the user experience.

[1609] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1610] In this invention, the server includes: a prompt or image input means for a user to specify a desired object; a generation AI generating code based on the specified prompt or image; a 3D model creation software creating an object using the generated code; a virtual environment placing the created object; and a sentiment analysis engine recognizing the user's sentiment and adjusting the responses and behavior of a virtual character based on the recognition results. This allows users to easily create and place 3D objects without requiring specialized knowledge, and enables natural and effective dialogue within the virtual environment utilizing sentiment analysis.

[1611] A "prompt" is input data that describes the target or condition desired by the user in natural language.

[1612] An "image" is digital data that visually represents an object or condition specified by a user.

[1613] "Generative AI" is an artificial intelligence system that analyzes user input data and automatically generates the necessary code and models.

[1614] "Code" refers to the program instructions generated by the AI ​​based on prompts and images specified by the user.

[1615] "3D model creation software" is software that actually generates 3D objects based on generated code.

[1616] A "virtual environment" is a computer-generated simulated space in which a user can interact.

[1617] A "virtual character" is an NPC (non-player character) in a virtual environment, a character that is set up to converse and interact with the user.

[1618] An "emotion analysis engine" is a technology that analyzes a user's voice, facial expressions, etc. to identify their emotional state.

[1619] "User's computer" means an information processing device used by a user to receive data from the generated AI and use it within the virtual environment.

[1620] An "information processing device" is a device that transmits, receives, analyzes, and stores digital data, and examples of this include servers.

[1621] The present invention provides a system that allows a user to easily create desired objects and place them in a virtual environment. Specific embodiments of the system will be described below.

[1622] Hardware and software used

[1623] This system mainly uses the following hardware and software:

[1624] Server: Receives prompts and image data and sends them to the generation AI

[1625] Terminal: A computer operated by a user, with 3D modeling software such as Blender installed.

[1626] Generative AI model: Analyzes user input data and generates the necessary Python scripts

[1627] Virtual environment management software: Place and manage objects and virtual characters in a virtual environment

[1628] Emotion Analysis Engine: Analyzes user emotions and adjusts the virtual character's responses and behavior

[1629] Detailed System Description

[1630] 1. Object creation prompt input

[1631] A user opens a browser and logs into the Metaverse admin page, where they select the "Create Object" option and enter the desired object's characteristics and shape as a prompt, or upload an existing image (e.g., "Wooden Bench").

[1632] Example prompt sentence:

[1633] "Generate a wooden bench."

[1634] "Create a virtual character that acts as a meeting advisor."

[1635] 2. Generate prompts and send them to the AI

[1636] The server receives user input and provides prompts and image data to the generative AI, which then creates a Python script to generate the object. This script contains commands and parameters that can be executed in 3D modeling software such as Blender.

[1637] 3. Generate and run the Python script

[1638] The AI ​​analyzes the prompt and automatically generates a Python script, which is then sent to the user's device via a server. The device then uses 3D modeling software such as Blender to generate objects based on the script.

[1639] 4. Placing objects in the virtual environment

[1640] The device places the generated 3D model in the virtual environment, and the user can adjust the object's exact position using the mouse or keyboard. Information about the placed object is registered in the virtual environment management software and shared with other users.

[1641] 5. Virtual Character Generation and Emotional Response

[1642] The server uses AI models to generate virtual characters, which can act as meeting advisors, idea-boosting assistants, etc. These characters are sent to the device based on the data generated by the AI ​​and placed in the virtual environment.

[1643] In addition, an emotion analysis engine analyzes the user's voice and facial expressions in real time and adjusts the virtual character's responses and behavior accordingly. For example, if the user looks unhappy, the virtual character will offer words of encouragement or suggest solutions to the problem.

[1644] Specific examples

[1645] For example, consider the case of generating a "wooden bench." The user selects "Object Creation" on the metaverse management screen and enters "wooden bench" as a prompt. The server sends this prompt to the generation AI, which creates a Python script. The server then sends the script to the device, which then runs it in Blender. The resulting 3D model of the "wooden bench" is placed in the lobby of the virtual office.

[1646] In addition, a virtual character called a "meeting advisor" is generated, and when a user interacts with the virtual environment, an emotion analysis engine analyzes the user's emotional state in real time and adjusts the virtual character's responses accordingly, allowing users to experience more natural and effective interactions within the virtual environment.

[1647] The above is a concrete example of how to implement the invention. This system allows users to easily generate 3D objects without requiring specialized knowledge, and enables effective dialogue utilizing emotion analysis.

[1648] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1649] Step 1:

[1650] The user opens a browser and accesses the metaverse management screen. The login screen is displayed, the user enters their username and password, and clicks the login button. The entered username and password are sent to the server for authentication. If authentication is successful, the metaverse management screen is displayed.

[1651] Input: Username, Password

[1652] Output: Metaverse admin page

[1653] Step 2:

[1654] On the Metaverse Admin screen, select the "Create Object" option. The user enters the desired object's characteristics and shape as a prompt, or uploads an existing image. For example, enter the prompt "Wooden bench." The user's input is sent to the server.

[1655] Input: A prompt or image (e.g., "Wooden bench")

[1656] Output: Prompt data sent to the server

[1657] Step 3:

[1658] The server takes the user's input and sends it to a generative AI, which parses the prompt and generates a Python script containing commands and parameters that can be executed in 3D modeling software such as Blender.

[1659] Input: prompt data

[1660] Output: Python script

[1661] Step 4:

[1662] The generated Python script is sent from the server to the device, where it is executed in 3D modeling software such as Blender. The specified object is generated based on the script.

[1663] Input: Python script

[1664] Output: 3D object

[1665] Step 5:

[1666] The device places the generated 3D object in the virtual environment. The user can adjust the object's position using the mouse or keyboard. The information about the placed object is saved in the virtual environment management software and can be shared with other users.

[1667] Input: 3D object

[1668] Output: Objects placed in the virtual environment

[1669] Step 6:

[1670] The server uses an AI model to generate a virtual character, which can act as a meeting advisor or idea-boosting assistant, for example. The generated character data is sent from the server to the device, which then places it in the virtual environment.

[1671] Input: Virtual character role (e.g., conference advisor)

[1672] Output: Generated virtual character data

[1673] Step 7:

[1674] The device places a virtual character in the virtual environment. When a user logs into the virtual environment as an avatar, they can begin interacting with the virtual character. An emotion analysis engine analyzes the user's emotions in real time and adjusts the virtual character's responses and behavior based on the results.

[1675] Input: Virtual character data, user emotion data

[1676] Output: User interaction with virtual character

[1677] Through the above process, this system allows users to easily create and place desired objects and effectively interact with them in the virtual environment.

[1678] (Application example 2)

[1679] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1680] Unlike real stores, creating product displays and layouts in virtual stores can be non-intuitive and require specialized skills, so there is a demand for technology that allows non-experts to easily create and arrange displays. Furthermore, there is a lack of means to optimize interactions by reflecting human emotions, which can degrade the quality of the user experience. Therefore, there is a demand for a system that can easily and effectively generate objects in virtual spaces and optimize interactions through emotion recognition.

[1681] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for a user to specify a desired object using a prompt or an image; means for a generation AI to generate a Python script based on the specified prompt or image; means for creating an object using 3D modeling software using the generated Python script; means for placing the created object in a virtual space; emotion recognition means for analyzing the user's emotions in real time; and means for adjusting the responses of NPCs based on the emotion recognition results. This allows even non-experts to intuitively generate and place objects in a virtual space, and further optimizes interactions according to the user's emotions, enabling a high-quality user experience.

[1682] A "prompt" is text or a command that a user enters into a computer system to instruct or request something.

[1683] An "image" is digital data representing visual information that can be displayed and processed on a computer.

[1684] "Generative AI" refers to artificial intelligence technology that automatically generates new data (in this case, Python scripts) based on input data.

[1685] A "Python script" is a series of instructions or procedures written in the Python programming language.

[1686] "3D modeling software" refers to software for creating and editing three-dimensional objects and scenes on a computer.

[1687] A "virtual space" is a computer-generated digital environment or simulation.

[1688] "Emotion recognition means" refers to technology or systems that analyze a user's facial expressions and voice data to identify their emotional state.

[1689] "NPC" stands for "Non-Player-Character" and refers to a virtual character that is not controlled by the user but can be interacted with.

[1690] MODE FOR CARRYING OUT THE INVENTION

[1691] The following describes an embodiment of the present invention. The system of the present invention allows a user to specify a desired object using a prompt or an image, automatically generates a Python script using generative AI, creates the object using 3D modeling software, and places the object in a virtual space. Furthermore, an emotion engine that recognizes the user's emotions is used to optimize interactions within the virtual space.

[1692] Hardware and software used

[1693] The system is implemented using the following hardware and software:

[1694] Server: Hosts the generative AI model and emotion recognition engine, and processes prompts and image data.

[1695] User's device: This includes smart glasses, smartphones, head-mounted displays, etc., used by users to access the virtual space.

[1696] Generative AI model: Generates Python scripts based on user prompts and images.

[1697] 3D modeling software: Execute the script generated using Blender or similar software to generate a 3D model.

[1698] Emotion recognition engine: Analyzes the user's facial expressions and voice data in real time to provide appropriate interactions.

[1699] Data processing and calculation

[1700] The server receives prompts and image data entered by the user and sends them to the generative AI model, which then generates a Python script based on the input and sends it back to the server. The server then sends the Python script to the user's device, which then runs the script in the 3D modeling software.

[1701] The generated 3D model is placed in the virtual space on the user's device. The device is also equipped with an emotion recognition engine that analyzes the user's emotional data in real time. Based on this emotional data, the server adjusts the responses and behavior of the NPC (virtual character) to provide a satisfying interaction for the user.

[1702] Specific examples

[1703] For example, a user puts on smart glasses and accesses a virtual store management screen, selects an empty display shelf in the store, and enters the prompt to "create a new display" or uploads an image of an existing display.

[1704] The server receives user input and sends prompts and image data to the generative AI model, which generates a suitable Python script in response to the prompt "Create a new display." This script is then sent from the server to the user's smart glasses, where it is generated and placed as a 3D object on a virtual display within the glasses.

[1705] The user checks the generated display in the virtual store and fine-tunes its placement as needed. The emotion recognition engine reads the user's facial expressions and voice to analyze their emotional state. For example, if the user shows a satisfied expression, the emotion engine recognizes this and notifies the user that the generation process was successful.

[1706] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1707] Step 1:

[1708] Users access the virtual store management screen using devices such as smart glasses or smartphones, where they can enter a prompt to create a desired object (e.g., "Create a new display") or upload an image of an existing display. The prompt or image data is provided as input.

[1709] Step 2:

[1710] The server receives prompts and image data entered by the user and sends them to the generative AI model. Specifically, the prompts and image data are used as input, and the generative AI model analyzes them. Here, preprocessing and tokenization of the data are performed.

[1711] Step 3:

[1712] The generative AI model generates a Python script based on the received prompt and image data. The generated Python script is output. The generative AI model automatically creates a script based on the specified prompt.

[1713] Step 4:

[1714] The server sends the generated Python script to the user's terminal. This is the process by which the generated script is transferred from the server to the user's terminal. The output from the server is a Python script, and the input to the terminal is the same script.

[1715] Step 5:

[1716] The user's device executes the received Python script in 3D modeling software (such as Blender) to generate a 3D object. In this step, the script is actually executed and a 3D model is generated. The input is the Python script, and the output is the 3D model.

[1717] Step 6:

[1718] The user's device places the generated 3D model in a virtual space. Specifically, the 3D model is placed in a virtual store as a virtual display. The input is the 3D model, and the output is placement in the virtual space.

[1719] Step 7:

[1720] The emotion recognition engine installed in the device analyzes the user's facial expressions and voice in real time to recognize their emotional state. The input is the user's facial expression data and voice data, and the emotional state is recognized as data calculation. Emotional data based on the analysis is output.

[1721] Step 8:

[1722] The server adjusts the NPC's responses and behavior based on the emotional data obtained from the emotion recognition engine. The input is the output data of the emotion recognition engine, and the server processes the data to adjust the NPC's responses and behavior. The adjusted NPC settings are obtained as the output.

[1723] Step 9:

[1724] The user interacts with the generated displays and NPCs in the virtual space and fine-tunes their placement as needed. This step optimizes the user experience. Specifically, the user can view the generated displays and receive feedback based on emotion recognition. The input is the user's behavior and emotion data, and the output is an optimized user experience.

[1725] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1726] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1727] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1728] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1729] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1730] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1731] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1732] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1733] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1734] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1735] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1736] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1737] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1738] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1739] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1740] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1741] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1742] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1743] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1744] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1745] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1746] The following is further disclosed regarding the above embodiment.

[1747] (Claim 1)

[1748] a means for the user to specify the desired object by prompt or image;

[1749] A means for the Generative AI to generate a Python script based on a given prompt or image;

[1750] a means for creating an object in 3D modeling software using the generated Python script;

[1751] A means for placing the created object in a virtual space;

[1752] A system including:

[1753] (Claim 2)

[1754] A means for users to participate in the virtual space as avatars;

[1755] A means for placing NPCs (Non-Player-Characters) in the virtual space and for the NPCs to interact with the user.

[1756] A means for users to refer to information generated by dialogue between NPCs;

[1757] The system of claim 1 further comprising:

[1758] (Claim 3)

[1759] a means for the generative AI to use a combination of image generation algorithms and 3D modeling algorithms to generate the object;

[1760] means for transmitting the generated 3D model to a user's terminal via a server;

[1761] A means for placing the transmitted 3D model in a virtual space on a user's terminal;

[1762] The system of claim 1 further comprising:

[1763] "Example 1"

[1764] (Claim 1)

[1765] an input means for a user to input a prompt or image;

[1766] A means for the AI ​​model to generate a Python script based on the input prompt or image; and

[1767] a means for creating an object in 3D modeling software using the generated Python script;

[1768] A means for placing the created object in a virtual space;

[1769] A system including:

[1770] (Claim 2)

[1771] A means for users to participate in the virtual space as avatars;

[1772] A means for generating and placing NPCs (Non-Player-Characters) in a virtual space and for the NPCs to interact with the user;

[1773] A means for users to refer to information generated by dialogue between NPCs;

[1774] 10. The system of claim 1, comprising:

[1775] (Claim 3)

[1776] a means for the generative AI model to use a combination of image generation algorithms and 3D modeling algorithms to generate the object;

[1777] means for transmitting the generated 3D model to a user's terminal via a server;

[1778] A means for placing the transmitted 3D model in a virtual space on a user's terminal;

[1779] 10. The system of claim 1, comprising:

[1780] "Application Example 1"

[1781] (Claim 1)

[1782] a means for the user to specify the desired object by prompt or image;

[1783] A means for the Generative AI to generate a program based on a specified prompt or image;

[1784] a means for creating an object in three-dimensional modeling software using the generated program;

[1785] A means for placing the created object in a virtual space;

[1786] A means for a user to view and operate the generated object in the virtual space;

[1787] A system including:

[1788] (Claim 2)

[1789] A means for users to participate in the virtual space as avatars;

[1790] A means for placing an automated character in a virtual space and for the automated character to interact with a user;

[1791] A means for a user to refer to information generated by the dialogue between the automated characters;

[1792] The system of claim 1 further comprising:

[1793] (Claim 3)

[1794] a means for the generative AI to use a combination of image generation algorithms and 3D modeling algorithms to generate the object;

[1795] means for transmitting the generated three-dimensional model to a user terminal via an information processing device;

[1796] A means for placing the transmitted three-dimensional model in a virtual space on a user's terminal;

[1797] The system of claim 1 further comprising:

[1798] "Example 2: Combining Emotion Engines"

[1799] (Claim 1)

[1800] a prompt or image input means for the user to specify a desired object;

[1801] A means for the AI ​​to generate code based on a given prompt or image; and

[1802] a means for creating an object in 3D modeling software using the generated code;

[1803] a means for placing the created object in a virtual environment;

[1804] A system including:

[1805] (Claim 2)

[1806] a means for a user to participate in the virtual environment as an avatar;

[1807] means for placing a virtual character within the virtual environment and for the virtual character to interact with a user;

[1808] A means for a user to refer to information generated by interactions between virtual characters;

[1809] means for recognizing a user's emotions using an emotion analysis engine and adjusting the responses and behavior of the virtual character based on the recognition results;

[1810] The system of claim 1 further comprising:

[1811] (Claim 3)

[1812] A means for the generating AI to use a combination of an image generation algorithm and a 3D model creation algorithm to generate an object;

[1813] means for transmitting the generated 3D model to a user's computer via an information processing device;

[1814] means for placing the transmitted 3D model in a virtual environment on the user's computer;

[1815] The system of claim 1 further comprising:

[1816] "Application example 2 when combining emotion engines"

[1817] (Claim 1)

[1818] a means for the user to specify the desired object by prompt or image;

[1819] A means for the Generative AI to generate a Python script based on a given prompt or image;

[1820] a means for creating an object in 3D modeling software using the generated Python script;

[1821] A means for placing the created object in a virtual space;

[1822] an emotion recognition means for analyzing the user's emotions in real time;

[1823] a means for adjusting the response of the NPC based on the emotion recognition result;

[1824] A system including:

[1825] (Claim 2)

[1826] A means for users to participate in the virtual space as avatars;

[1827] A means for placing NPCs (Non-Player-Characters) in the virtual space and for the NPCs to interact with the user.

[1828] A means for users to refer to information generated by dialogue between NPCs;

[1829] A means for adjusting NPC dialogue based on real-time emotion recognition results;

[1830] The system of claim 1 further comprising:

[1831] (Claim 3)

[1832] a means for the generative AI to use a combination of image generation algorithms and 3D modeling algorithms to generate the object;

[1833] means for transmitting the generated 3D model to a user's terminal via a server;

[1834] A means for placing the transmitted 3D model in a virtual space on a user's terminal;

[1835] The emotion recognition means analyzes facial expressions and voice data of a user;

[1836] The system of claim 1 further comprising: [Explanation of symbols]

[1837] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for the user to specify the desired object by prompt or image; A means for the Generative AI to generate a Python script based on a given prompt or image; a means for creating an object in 3D modeling software using the generated Python script; A means for placing the created object in a virtual space; A system including:

2. A means for users to participate in the virtual space as avatars; A means for placing NPCs (Non-Player-Characters) in the virtual space and for the NPCs to interact with the user. A means for users to refer to information generated by dialogue between NPCs; The system of claim 1 further comprising:

3. a means for the generative AI to use a combination of image generation algorithms and 3D modeling algorithms to generate the object; means for transmitting the generated 3D model to a user's terminal via a server; A means for placing the transmitted 3D model in a virtual space on a user's terminal; The system of claim 1 further comprising:

Images