system

Abstract

We provide the system. [Solution] A means of acquiring three-dimensional model data selected in a virtual environment, A means for converting the three-dimensional model data into a specific output format using artificial intelligence that generates the data, A means of transmitting the converted data to a three-dimensional printing device and outputting it as a physical object, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a virtual reality environment, although a user can visually experience three-dimensional objects, it is difficult to bring them back to the real world in a physical form. In particular, due to the lack of an efficient method for directly converting three-dimensional model data into a form that can be used in the real world, there is a problem that an end user cannot easily output an object that they want to take back from the virtual environment.

Means for Solving the Problems

[0005] The present invention solves the above problem by providing a system that automatically converts three-dimensional model data obtained in a virtual environment into an output format for a specific three-dimensional printing device using artificial intelligence. The system can seamlessly perform the process of efficiently collecting three-dimensional model data selected by the user in a virtual environment, converting it into an output-ready format, and finally outputting it into the real world as a physical object using a three-dimensional printing device.

[0006] A "virtual environment" is a digital three-dimensional space generated by a computer system, within which users can interact visually and aurally.

[0007] "Three-dimensional model data" refers to digital data used to represent the shape and structure of an object in three-dimensional space, and is used in virtual reality and three-dimensional printing.

[0008] "Artificial intelligence" refers to methods by which computer programs mimic learning, reasoning, pattern recognition, and other intelligent behaviors, and in this context, includes technologies used to transform three-dimensional data.

[0009] A "three-dimensional printing device" is a machine that can create physical objects in layers based on digital data, and is commonly known as a "3D printer."

[0010] "Output format" refers to the data structure and description method required for a three-dimensional printing device to physically produce the output. [Brief explanation of the drawing]

[0011] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3]This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0012] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0013] First, let's explain the terminology used in the following explanation.

[0014] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0015] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0016] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0017] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applicable to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), and the like.

[0018] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0019] [First Embodiment]

[0020] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0021] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0022] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0023] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0024] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0025] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0026] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0027] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0028] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0029] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0030] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0031] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0032] This invention provides a system that includes a program for generating a three-dimensional model selected by a user within a virtual environment. The system includes a terminal, a server, and a three-dimensional printing device. The specific processing of each component is described below.

[0033] The user selects a 3D object through a virtual reality interface on their device. This selection causes the device to retrieve the corresponding 3D model data and send it to a server. The server then analyzes the received data and uses artificial intelligence to automatically convert it into a specific output format usable by a 3D printer. The converted data is then sent from the server to the 3D printer and output as a physical object.

[0034] For example, if a user selects a sculpture they are interested in within a virtual exhibition and wishes to replicate its 3D model data in the real world, the terminal captures the selected data and sends it to a server. The server processes, transforms, and optimizes the data, then outputs the physical object using a designated 3D printer. This entire process allows users to easily and quickly bring objects from the virtual environment into the real world.

[0035] This system provides an efficient and user-friendly method for physically recreating any three-dimensional model that a user finds in virtual reality. Users can quickly achieve their goals without special technical knowledge, as automated processing and 3D printing are performed via the server through their terminal.

[0036] The following describes the processing flow.

[0037] Step 1:

[0038] The user selects 3D objects within a virtual environment. The user visually confirms and selects objects of interest through a virtual reality interface.

[0039] Step 2:

[0040] The device captures the selected 3D model data. The device receives the user's selection and temporarily saves the 3D model data in its internal memory.

[0041] Step 3:

[0042] The device sends 3D model data to the server. The device sends the saved data to the server via the internet connection and verifies that it was sent correctly.

[0043] Step 4:

[0044] The server analyzes the received data. The server checks the incoming 3D model data and determines whether the conversion process can proceed without problems.

[0045] Step 5:

[0046] The server uses artificial intelligence to transform the data. The server passes the analyzed data to a generating AI, which then converts it into a specific output format that can be used by a 3D printing device.

[0047] Step 6:

[0048] The server verifies the converted data. The server checks the integrity of the output data and optimizes it to ensure data quality.

[0049] Step 7:

[0050] The server sends the converted and optimized data to the 3D printing device. The server then sends the final data to the designated printing device via the internet.

[0051] Step 8:

[0052] The 3D printing device creates a physical object based on the transmitted data. Upon completion, the user receives a notification that they can receive the physical object.

[0053] (Example 1)

[0054] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0055] Traditional 3D printing processes required advanced technical knowledge and complex procedures to reproduce a 3D model selected by the user in a virtual environment in the real world. Furthermore, data conversion and print optimization were often performed manually, which was time-consuming and labor-intensive, making it difficult to generate objects quickly and intuitively.

[0056] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0057] In this invention, the server includes means for acquiring data of an object model selected in a virtual environment via a terminal operated by the user, means for analyzing the data of the object model and converting it into a specific printable format using a machine learning algorithm, and means for transmitting the converted data to a printing device and materializing it as an object. This makes it possible for the user to quickly and intuitively reproduce a model in the virtual environment physically without requiring any special technical knowledge.

[0058] A "terminal" is an electronic device that a user operates to select objects within a virtual environment and transmit that selection information to a server.

[0059] A "server" is a computer system that receives data sent from a terminal, analyzes and converts it, and then sends it to a printing device.

[0060] An "object model" is digital data of a three-dimensional shape that can be selected by the user within a virtual environment.

[0061] A "machine learning algorithm" is an artificial intelligence computation method used to analyze data and convert it into a specific format.

[0062] A "printing device" is equipment used to generate physical objects based on converted digital data.

[0063] "Conversion" is the process of automatically changing the data of an object model into a printable format.

[0064] "Optimization" refers to adjusting data and processes to perform printing efficiently.

[0065] "Manifestation" refers to the process of outputting model data from a virtual environment in a physical form.

[0066] This invention is a system comprising a user-operated terminal, a server for analyzing and converting data, and a printing device for generating physical objects. The user selects a three-dimensional object of interest using a virtual reality interface on the terminal. The terminal acquires the digital data of the selected three-dimensional model and transmits that data to the server. The server utilizes a generated AI model to analyze and convert the model data, automatically converting it into a format usable by the printing device (e.g., G-code).

[0067] For example, if a user wants to recreate a specific sculpture they saw in a virtual museum, they select the sculpture on their device. The device then captures digital data and sends it to a server. The server uses a generative AI model to convert the data into the appropriate format and sends it to a printing device, thereby generating the physical sculpture.

[0068] In this invention, an example of a generated prompt message might be an instruction such as, "Please convert the selected three-dimensional model into a printable format." This system helps users easily reproduce a model selected in a virtual environment as a real object, even without special technical knowledge.

[0069] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0070] Step 1:

[0071] The user selects a 3D object of interest using the virtual reality interface on the terminal. The input is the object ID selected by the user in the virtual space. Based on this selection, the terminal retrieves the corresponding 3D model data and sends it to the server. The output is the 3D model data sent to the server.

[0072] Step 2:

[0073] The server receives 3D model data sent from the terminal. The input is the 3D model data sent from the terminal. The server analyzes the data using a generative AI model and performs data processing to convert it into a specific printable format (e.g., G-code). The output is the data converted into a format usable by a printing device.

[0074] Step 3:

[0075] The server sends the converted data to the printer. The input is the printable data converted by the server. The server sends this data to the printer and performs actions to instruct it to materialize an object. The output is the physical object produced by the printer.

[0076] This process allows users to quickly and intuitively recreate selected three-dimensional objects in the real world.

[0077] (Application Example 1)

[0078] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0079] There is a need to efficiently reproduce three-dimensional representations selected in a virtual space using visual input devices as physical objects in the real world, by modifying them to the shape desired by the user. Conventional methods often require advanced expertise and complex operations to convert data selected and modified by the user in the virtual space into physical objects, making them difficult for the average user.

[0080] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0081] In this invention, the server includes means for acquiring three-dimensional representation data selected in a virtual space, means for converting the three-dimensional representation data into a specific output format using a machine learning model that generates the data, means for transmitting the converted data to a manufacturing device and outputting it as a physical item, and means for acquiring and reflecting selection and modification information of items in the virtual space using a visual input device. This makes it possible for users to efficiently and quickly materialize items they have selected and modified in the real world from the virtual space without requiring specialized knowledge.

[0082] A "virtual space" is a computer-generated environment that users can visually experience through digital technology.

[0083] "Three-dimensional representation data" refers to digital data that contains shape information constructed in three dimensions, and is a concrete representation of the shape that is visually experienced in a virtual space.

[0084] A "machine learning model" is a type of algorithm that learns from large amounts of data and transforms or recognizes input data to suit a specific purpose.

[0085] An "output format" is a standard for representing data obtained through machine processing in a physically or digitally defined format.

[0086] "Manufacturing equipment" refers to devices used to create physical objects or structures based on digital data, and three-dimensional printers are a prime example of this.

[0087] A "visual input device" is a device that acquires a user's visual information and processes it digitally, and includes, for example, smart glasses.

[0088] This invention provides a system and process for materializing objects selected in a virtual space as physical shapes in the real world. Specific embodiments thereof are described below.

[0089] The user uses smart glasses, a visual input device, to select any 3D representation in a virtual space. The user can also modify the color, size, and material of that 3D representation. This information is transmitted to the terminal in real time via edge computing.

[0090] The terminal sends the acquired 3D representation data to a cloud server. This server is equipped with a machine learning model using the TENSORFLOW® library, which analyzes the transmitted data and converts it into an output format usable by the manufacturing equipment. The conversion result is then sent to the manufacturing equipment.

[0091] The manufacturing equipment includes 3D printers that produce physical objects based on digital data. These 3D printers quickly and accurately output objects selected and modified by the user in a virtual space based on the received data.

[0092] As a concrete example, consider a scenario where a user selects a blue chair in a virtual store and adjusts its size. The 3D representation data selected through smart glasses is quickly processed on a cloud server and output by a designated 3D printer. This allows the user to receive a real-world object that directly reflects their visual selection in the virtual space.

[0093] An example of a prompt for a generated AI model is, "The selected 3D representation is a blue chair. Adjust the size and convert it into data for printing on a real-world 3D printer." This prompt serves to instruct the server to perform the optimal data conversion.

[0094] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0095] Step 1:

[0096] The user uses smart glasses, a visual input device, to select a 3D representation within a virtual space. Sensors in the glasses detect the user's gaze and gesture input, and transmit this selection information to the terminal via edge computing. The input at this time is the selected 3D representation data, and the output is the transmission of that data.

[0097] Step 2:

[0098] The terminal temporarily stores the received 3D representation data and then sends it to the cloud server. This data transfer, based on user input, is performed quickly and securely. Furthermore, data checks are performed using a communication protocol to ensure error-free transmission. The input is the stored 3D representation data, and the output is sending this data to the cloud server.

[0099] Step 3:

[0100] The server runs a machine learning model using the TensorFlow library on the cloud and analyzes the received 3D representation data. The analysis results are converted into a specific output format usable by 3D printers. The input is the transmitted 3D representation data, and the output is the data converted into the specific output format. The analysis also includes data correction and noise reduction using a generative AI model.

[0101] Step 4:

[0102] The server sends the converted data to the manufacturing equipment, specifically a 3D printer. This transmission uses network communication, and the server also checks the status of the receiving manufacturing equipment (running, stopped, etc.). The input is the converted output format data, and the output is the data transmission to the 3D printer.

[0103] Step 5:

[0104] The 3D printer, a manufacturing device, begins printing a physical object based on the received data. The printer is calibrated, and the material selection and layer configuration of the object to be printed are optimized. The input is data sent from the server, and the output is the completed physical object.

[0105] This processing flow allows users to materialize selected objects in the virtual world as physical objects in the real world. Because this series of operations is performed quickly, users can experience a seamless transformation from digital to physical.

[0106] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0107] This invention relates to a system that recognizes a user's emotions in a virtual environment and recommends and outputs an optimal three-dimensional model based on those emotions. The system includes a terminal, a server, a three-dimensional printing device, and an emotion engine that analyzes the user's emotions.

[0108] When a user engages in activities within the virtual environment, the device has a function that senses the user's facial expressions and voice input, and based on this, an emotion engine analyzes the user's emotional state. The emotion engine determines the user's emotions in real time and provides a selection of 3D models that match those emotions.

[0109] When a user accepts an emotion-based recommendation and selects a specific 3D object, the terminal confirms the selection and transfers the 3D model data to the server. The server has the capability to analyze the data and convert it into a format suitable for a 3D printer using artificial intelligence. After conversion, the server sends the data to the 3D printer and executes the process of outputting it as a physical object.

[0110] For example, if the emotion engine determines that a user is feeling stressed, it can recommend objects with a relaxing effect, such as a smooth-shaped ornament. If the user selects this, it will materialize in the real world as a design that promotes relaxation.

[0111] By incorporating an emotional engine, the system can provide flexible services that adapt to the user's current mental and physical state. This allows users to go beyond a mere visual experience and enhance their emotional satisfaction.

[0112] The following describes the processing flow.

[0113] Step 1:

[0114] The user logs into the virtual environment and begins their activities. The user interacts with the interface to explore multiple visually presented three-dimensional objects.

[0115] Step 2:

[0116] The device collects emotional data from the user's facial expressions and voice. The device uses built-in sensors to monitor the user's facial movements and voice tone in real time and transmits the data to the emotion engine.

[0117] Step 3:

[0118] The emotion engine analyzes the received data to identify the user's current emotional state. The emotion engine uses algorithms to determine the user's emotions, such as stress, joy, and surprise.

[0119] Step 4:

[0120] The emotion engine generates a list of recommended 3D objects based on the user's emotions and sends it to the terminal. The terminal then presents this list to the user via a control panel or overlay display within the virtual environment.

[0121] Step 5:

[0122] The user selects a 3D object of interest from emotion-based recommendations. The user confirms their selection by pointing to the object or pressing a select button.

[0123] Step 6:

[0124] The terminal sends the selected 3D model data to the server. The terminal securely uploads the data to the server via the internet connection and waits for confirmation of receipt.

[0125] Step 7:

[0126] The server receives the 3D model data and uses AI to convert it into a printable format. The server then checks the quality and output suitability of the converted data.

[0127] Step 8:

[0128] After the server completes the conversion, it sends the data to the 3D printer and begins printing the physical object. The server monitors the printing progress and sends a completion notification to the terminal.

[0129] Step 9:

[0130] Users receive notification via their device when printing is complete and can pick up the physical object at a designated location. Ultimately, users can view the object in the real world and use or display it.

[0131] (Example 2)

[0132] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0133] The challenge lies in providing a system that can enhance the user experience by offering flexibly and effectively customized 3D models based on user emotions. Such a system is required to reflect the user's emotional state in real time and improve emotional and visual satisfaction by generating physical objects appropriate to that state.

[0134] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0135] In this invention, the server includes means for analyzing sensor information to detect the user's emotions, means for presenting a selection of three-dimensional models based on the detected emotion information, and means for acquiring the selected three-dimensional model data in a virtual environment. This makes it possible to propose and generate three-dimensional models that are in line with the user's emotional state.

[0136] "Emotion detection" means analyzing the user's facial expressions and voice to understand their psychological state in real time.

[0137] "Analyzing sensor information" is the process of estimating a user's physiological and psychological state using data acquired from input devices such as cameras and microphones.

[0138] A "three-dimensional model" is three-dimensional object data generated on a computer that possesses a shape that can be physically output.

[0139] "Presenting options" means showing the user a list of three-dimensional models that appear to be the most suitable based on the user's emotional information.

[0140] A "three-dimensional printing device" is a device that creates physical three-dimensional objects based on digital data.

[0141] "Artificial intelligence" refers to all methods and technologies for performing intelligent tasks using computer programs, and in this invention, it is specifically used for data analysis and generation.

[0142] This invention is a system that recognizes a user's emotions in a virtual environment and recommends and outputs an optimal three-dimensional model based on those emotions. The system comprises a terminal, a server, and a three-dimensional printing device.

[0143] The device uses input devices such as cameras and microphones to capture the user's facial expressions and voice in real time in order to detect the user's emotions. The acquired data is analyzed by an emotion analysis engine to identify the user's physiological and psychological state. Emotion analysis software is used for this analysis.

[0144] The server generates a selection of the most suitable 3D models for the user based on the analysis results from the emotion analysis engine. At this time, it utilizes the generated AI model to extract highly relevant 3D models from the database and presents them to the user.

[0145] The user selects their preferred 3D model from the options provided on the device. This selection information is sent from the device to the server. The generating AI model converts the selected 3D model data into the appropriate output format and prepares it for generation as a physical object in a 3D printer.

[0146] For example, if the emotion engine determines that the user is feeling stressed, it will recommend an object with a relaxing effect (for example, a smooth-shaped ornament). When the user selects this object, the terminal transmits the selection information to the server, which converts the data into a format suitable for a printer and outputs it as a physical object. An example of a prompt message would be, "The user is feeling stressed. Please suggest a 3D model with a relaxing effect."

[0147] This system allows users to obtain a three-dimensional model that reflects their emotional state, resulting in improved visual and emotional satisfaction.

[0148] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0149] Step 1:

[0150] The device uses its camera and microphone to capture the user's facial expressions and voice in real time. The input data includes emotional indicators such as facial movements, facial expressions, and voice tone. The device sends this data to an emotion analysis engine.

[0151] Step 2:

[0152] The emotion analysis engine analyzes the data transmitted from the terminal. Specifically, emotion analysis software is used to estimate the user's psychological state (e.g., joy, sadness, stress) from the data. The output of this process is information indicating the estimated emotional state of the user.

[0153] Step 3:

[0154] The server receives emotional state information obtained from the emotion analysis engine as input. Based on this information, it utilizes a generative AI model to generate a selection of 3D models suitable for the emotion. This output is, for example, a list of 3D models with shapes that promote relaxation.

[0155] Step 4:

[0156] The user reviews the 3D model options presented on the device and selects their desired model. The user's selection action is recorded as input on the device, and the selected model information is sent from the device to the server.

[0157] Step 5:

[0158] After receiving the selected model information from the terminal, the server uses a generated AI model to convert the data into an appropriate output format for the 3D printing device. This conversion process includes format conversion and scaling adjustments. The converted data is then obtained as output.

[0159] Step 6:

[0160] The server sends the converted data to the 3D printer. The 3D printer uses this data to begin the process of generating the actual physical object. Specifically, it uses materials to sequentially construct the shape.

[0161] (Application Example 2)

[0162] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0163] Modern consumers demand personalized experiences with goods, but traditional systems struggle to provide this in real time. Furthermore, there's a lack of technology to instantly suggest and physically deliver special products that cater to consumer emotions and preferences. This hinders improvements in customer satisfaction in retail stores.

[0164] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0165] In this invention, the server includes means for analyzing the user's emotional state in a virtual space and recommending a three-dimensional model that is individually suited to the user; means for converting the three-dimensional model data into a specific output format using artificial intelligence; and means for transmitting the converted data to a three-dimensional printing device and outputting it as a physical structure. This makes it possible to provide personalized goods based on the consumer's emotions.

[0166] A "virtual space" is an artificial environment created by a computer, a space in which users can immerse themselves and experience various sensory sensations.

[0167] "User's emotional state" refers to an individual's psychological or emotional condition, which is analyzed from facial expressions, voice, and other factors.

[0168] A "three-dimensional model" is a model of a three-dimensional shape or structure, represented physically or digitally.

[0169] "Artificial intelligence" is a general term for human intelligent behavioral processes developed with the aim of being imitated by computers.

[0170] A "specific output format" refers to a format or style that is suitable for a particular purpose or use.

[0171] A "3D printing device" is a machine that materializes physical objects by constructing them layer by layer based on three-dimensional design data.

[0172] A "physical structure" is a real-world object or shape that can be touched in actual space.

[0173] A "person of interest" refers to an individual or group that shows interest in a particular service or product.

[0174] "Emotion-based recommendations" refer to suggestions or recommendations that reflect an individual's psychological state.

[0175] A "buying space" refers to a physical or digital marketplace where products and services are traded.

[0176] In embodiments of the present invention, the system consists of a user, a terminal, a server, and a 3D printing device. The user accesses a virtual space through a dedicated terminal and engages in activities within it through actions, facial expressions, and voice. The terminal senses the user's facial expressions and voice in real time and analyzes the user's emotional state using an emotion analysis engine.

[0177] At this stage, the terminal presents the user with appropriate 3D model options based on the analyzed emotions. If the user selects a specific 3D model, that information is transferred to the server. The server uses a generative AI model to convert the 3D model data into a specific output format and sends that data to a 3D printing device to output it as a physical structure.

[0178] For example, if the device analyzes that the user is in an emotional state seeking relaxation, it will recommend a shape aimed at stress reduction, such as a rounded object. If the user selects this three-dimensional shape, the model is output by a 3D printer via the server and provided to the user in real time.

[0179] In this process, the generative AI model assists in automatically generating appropriate models by using prompts such as, "If the user's emotion is one of relaxation, what kind of 3D model design should be proposed?"

[0180] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0181] Step 1:

[0182] The device acquires the user's facial expressions and voice data. This data is input collected in real time through the camera and microphone. The data obtained is then analyzed by an emotion analysis engine, and the user's emotional state is output.

[0183] Step 2:

[0184] The device presents the user with appropriate 3D model options based on the output of its emotion analysis engine. If the emotional state is relaxed, a 3D model with a smooth shape is suggested. A generative AI model is used for the suggestions, and the prompt "What 3D model should be presented based on the emotion?" is input, and a list of appropriate models is output.

[0185] Step 3:

[0186] The user selects their preferred 3D model from those presented. This selection is entered into the terminal and transmitted to the server. The user's selection information is sent to the server as input data, and this information is used in the next conversion process.

[0187] Step 4:

[0188] The server converts the user's selected 3D model data into a specific output format. This process again utilizes a generative AI model to adjust the model data into a format suitable for a 3D printer. The input is the user's selected model data, and the output is data usable by the 3D printer.

[0189] Step 5:

[0190] The converted output data is sent from the server to the 3D printing machine. The printing machine receives this data and outputs a model as a physical structure. Once the process from data transmission to model printing is complete, it is finally provided to the user.

[0191] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0192] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0193] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0194] [Second Embodiment]

[0195] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0196] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0197] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0198] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0199] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0200] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0201] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0202] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0203] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0204] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0205] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0206] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0207] This invention provides a system that includes a program for generating a three-dimensional model selected by a user within a virtual environment. The system includes a terminal, a server, and a three-dimensional printing device. The specific processing of each component is described below.

[0208] The user selects a 3D object through a virtual reality interface on their device. This selection causes the device to retrieve the corresponding 3D model data and send it to a server. The server then analyzes the received data and uses artificial intelligence to automatically convert it into a specific output format usable by a 3D printer. The converted data is then sent from the server to the 3D printer and output as a physical object.

[0209] For example, if a user selects a sculpture they are interested in within a virtual exhibition and wishes to replicate its 3D model data in the real world, the terminal captures the selected data and sends it to a server. The server processes, transforms, and optimizes the data, then outputs the physical object using a designated 3D printer. This entire process allows users to easily and quickly bring objects from the virtual environment into the real world.

[0210] This system provides an efficient and user-friendly method for physically recreating any three-dimensional model that a user finds in virtual reality. Users can quickly achieve their goals without special technical knowledge, as automated processing and 3D printing are performed via the server through their terminal.

[0211] The following describes the processing flow.

[0212] Step 1:

[0213] The user selects 3D objects within a virtual environment. The user visually confirms and selects objects of interest through a virtual reality interface.

[0214] Step 2:

[0215] The device captures the selected 3D model data. The device receives the user's selection and temporarily saves the 3D model data in its internal memory.

[0216] Step 3:

[0217] The device sends 3D model data to the server. The device sends the saved data to the server via the internet connection and verifies that it was sent correctly.

[0218] Step 4:

[0219] The server analyzes the received data. The server checks the incoming 3D model data and determines whether the conversion process can proceed without problems.

[0220] Step 5:

[0221] The server uses artificial intelligence to transform the data. The server passes the analyzed data to a generating AI, which then converts it into a specific output format that can be used by a 3D printing device.

[0222] Step 6:

[0223] The server verifies the converted data. The server checks the integrity of the output data and optimizes it to ensure data quality.

[0224] Step 7:

[0225] The server sends the converted and optimized data to the 3D printing device. The server then sends the final data to the designated printing device via the internet.

[0226] Step 8:

[0227] The 3D printing device creates a physical object based on the transmitted data. Upon completion, the user receives a notification that they can receive the physical object.

[0228] (Example 1)

[0229] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0230] Traditional 3D printing processes required advanced technical knowledge and complex procedures to reproduce a 3D model selected by the user in a virtual environment in the real world. Furthermore, data conversion and print optimization were often performed manually, which was time-consuming and labor-intensive, making it difficult to generate objects quickly and intuitively.

[0231] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0232] In this invention, the server includes means for acquiring data of an object model selected in a virtual environment via a terminal operated by the user, means for analyzing the data of the object model and converting it into a specific printable format using a machine learning algorithm, and means for transmitting the converted data to a printing device and materializing it as an object. This makes it possible for the user to quickly and intuitively reproduce a model in the virtual environment physically without requiring any special technical knowledge.

[0233] A "terminal" is an electronic device that a user operates to select objects within a virtual environment and transmit that selection information to a server.

[0234] A "server" is a computer system that receives data sent from a terminal, analyzes and converts it, and then sends it to a printing device.

[0235] An "object model" is digital data of a three-dimensional shape that can be selected by the user within a virtual environment.

[0236] A "machine learning algorithm" is an artificial intelligence computation method used to analyze data and convert it into a specific format.

[0237] A "printing device" is equipment used to generate physical objects based on converted digital data.

[0238] "Conversion" is the process of automatically changing the data of an object model into a printable format.

[0239] "Optimization" refers to adjusting data and processes to perform printing efficiently.

[0240] "Manifestation" refers to the process of outputting model data from a virtual environment in a physical form.

[0241] This invention is a system comprising a user-operated terminal, a server for analyzing and converting data, and a printing device for generating physical objects. The user selects a three-dimensional object of interest using a virtual reality interface on the terminal. The terminal acquires the digital data of the selected three-dimensional model and transmits that data to the server. The server utilizes a generated AI model to analyze and convert the model data, automatically converting it into a format usable by the printing device (e.g., G-code).

[0242] For example, if a user wants to recreate a specific sculpture they saw in a virtual museum, they select the sculpture on their device. The device then captures digital data and sends it to a server. The server uses a generative AI model to convert the data into the appropriate format and sends it to a printing device, thereby generating the physical sculpture.

[0243] In this invention, an example of a generated prompt message might be an instruction such as, "Please convert the selected three-dimensional model into a printable format." This system helps users easily reproduce a model selected in a virtual environment as a real object, even without special technical knowledge.

[0244] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0245] Step 1:

[0246] The user selects a 3D object of interest using the virtual reality interface on the terminal. The input is the object ID selected by the user in the virtual space. Based on this selection, the terminal retrieves the corresponding 3D model data and sends it to the server. The output is the 3D model data sent to the server.

[0247] Step 2:

[0248] The server receives 3D model data sent from the terminal. The input is the 3D model data sent from the terminal. The server analyzes the data using a generative AI model and performs data processing to convert it into a specific printable format (e.g., G-code). The output is the data converted into a format usable by a printing device.

[0249] Step 3:

[0250] The server sends the converted data to the printer. The input is the printable data converted by the server. The server sends this data to the printer and performs actions to instruct it to materialize an object. The output is the physical object produced by the printer.

[0251] This process allows users to quickly and intuitively recreate selected three-dimensional objects in the real world.

[0252] (Application Example 1)

[0253] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0254] There is a need to efficiently reproduce three-dimensional representations selected in a virtual space using visual input devices as physical objects in the real world, by modifying them to the shape desired by the user. Conventional methods often require advanced expertise and complex operations to convert data selected and modified by the user in the virtual space into physical objects, making them difficult for the average user.

[0255] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0256] In this invention, the server includes means for acquiring three-dimensional representation data selected in a virtual space, means for converting the three-dimensional representation data into a specific output format using a machine learning model that generates the data, means for transmitting the converted data to a manufacturing device and outputting it as a physical item, and means for acquiring and reflecting selection and modification information of items in the virtual space using a visual input device. This makes it possible for users to efficiently and quickly materialize items they have selected and modified in the real world from the virtual space without requiring specialized knowledge.

[0257] A "virtual space" is a computer-generated environment that users can visually experience through digital technology.

[0258] "Three-dimensional representation data" refers to digital data that contains shape information constructed in three dimensions, and is a concrete representation of the shape that is visually experienced in a virtual space.

[0259] A "machine learning model" is a type of algorithm that learns from large amounts of data and transforms or recognizes input data to suit a specific purpose.

[0260] An "output format" is a standard for representing data obtained through machine processing in a physically or digitally defined format.

[0261] "Manufacturing equipment" refers to devices used to create physical objects or structures based on digital data, and three-dimensional printers are a prime example of this.

[0262] A "visual input device" is a device that acquires a user's visual information and processes it digitally, and includes, for example, smart glasses.

[0263] This invention provides a system and process for materializing objects selected in a virtual space as physical shapes in the real world. Specific embodiments thereof are described below.

[0264] The user uses smart glasses, a visual input device, to select any 3D representation in a virtual space. The user can also modify the color, size, and material of that 3D representation. This information is transmitted to the terminal in real time via edge computing.

[0265] The terminal sends the acquired 3D representation data to a cloud server. This server is equipped with a machine learning model using the TensorFlow library, which analyzes the transmitted data and converts it into an output format usable by the manufacturing equipment. The conversion results are then sent to the manufacturing equipment.

[0266] The manufacturing equipment includes 3D printers that produce physical objects based on digital data. These 3D printers quickly and accurately output objects selected and modified by the user in a virtual space based on the received data.

[0267] As a concrete example, consider a scenario where a user selects a blue chair in a virtual store and adjusts its size. The 3D representation data selected through smart glasses is quickly processed on a cloud server and output by a designated 3D printer. This allows the user to receive a real-world object that directly reflects their visual selection in the virtual space.

[0268] An example of a prompt for a generated AI model is, "The selected 3D representation is a blue chair. Adjust the size and convert it into data for printing on a real-world 3D printer." This prompt serves to instruct the server to perform the optimal data conversion.

[0269] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0270] Step 1:

[0271] The user uses smart glasses, a visual input device, to select a 3D representation within a virtual space. Sensors in the glasses detect the user's gaze and gesture input, and transmit this selection information to the terminal via edge computing. The input at this time is the selected 3D representation data, and the output is the transmission of that data.

[0272] Step 2:

[0273] The terminal temporarily stores the received 3D representation data and then sends it to the cloud server. This data transfer, based on user input, is performed quickly and securely. Furthermore, data checks are performed using a communication protocol to ensure error-free transmission. The input is the stored 3D representation data, and the output is sending this data to the cloud server.

[0274] Step 3:

[0275] The server runs a machine learning model using the TensorFlow library on the cloud and analyzes the received 3D representation data. The analysis results are converted into a specific output format usable by 3D printers. The input is the transmitted 3D representation data, and the output is the data converted into the specific output format. The analysis also includes data correction and noise reduction using a generative AI model.

[0276] Step 4:

[0277] The server sends the converted data to the manufacturing equipment, specifically a 3D printer. This transmission uses network communication, and the server also checks the status of the receiving manufacturing equipment (running, stopped, etc.). The input is the converted output format data, and the output is the data transmission to the 3D printer.

[0278] Step 5:

[0279] The 3D printer, a manufacturing device, begins printing a physical object based on the received data. The printer is calibrated, and the material selection and layer configuration of the object to be printed are optimized. The input is data sent from the server, and the output is the completed physical object.

[0280] This processing flow allows users to materialize selected objects in the virtual world as physical objects in the real world. Because this series of operations is performed quickly, users can experience a seamless transformation from digital to physical.

[0281] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0282] This invention relates to a system that recognizes a user's emotions in a virtual environment and recommends and outputs an optimal three-dimensional model based on those emotions. The system includes a terminal, a server, a three-dimensional printing device, and an emotion engine that analyzes the user's emotions.

[0283] When a user engages in activities within the virtual environment, the device has a function that senses the user's facial expressions and voice input, and based on this, an emotion engine analyzes the user's emotional state. The emotion engine determines the user's emotions in real time and provides a selection of 3D models that match those emotions.

[0284] When the user accepts a recommendation based on emotion and selects a specific three-dimensional object, the terminal confirms the selection and transfers the 3D model data to the server. The server has a function that can analyze the data and convert it into a format suitable for a three-dimensional printing device using artificial intelligence for generation. After the conversion, the server transmits the data to the three-dimensional printing device and executes the process of outputting it as a physical object.

[0285] As a specific example, when the emotion engine determines that the user is feeling stressed, it can recommend objects with a relaxation effect, such as ornaments with a smooth shape. If the user selects this, it will be embodied as a design that promotes relaxation in the real world.

[0286] By incorporating an emotion engine, the system can provide flexible services adapted to the user's current physical and mental state. As a result, the user can go beyond a mere visual experience and improve their emotional satisfaction.

[0287] The following describes the processing flow.

[0288] Step 1:

[0289] The user logs in to the virtual environment and starts activities. The user operates the interface to explore a plurality of three-dimensional objects presented visually.

[0290] Step 2:

[0291] The terminal collects emotion data from the user's expression and voice. The terminal uses built-in sensors to monitor the user's facial movements and voice tones in real time and transmits the data to the emotion engine. [[ID=第29]]

[0292] ステップ3:

[0293] The emotion engine analyzes the received data to identify the user's current emotional state. The emotion engine uses algorithms to determine the user's emotions, such as stress, joy, and surprise.

[0294] Step 4:

[0295] The emotion engine generates a list of recommended 3D objects based on the user's emotions and sends it to the terminal. The terminal then presents this list to the user via a control panel or overlay display within the virtual environment.

[0296] Step 5:

[0297] The user selects a 3D object of interest from emotion-based recommendations. The user confirms their selection by pointing to the object or pressing a select button.

[0298] Step 6:

[0299] The terminal sends the selected 3D model data to the server. The terminal securely uploads the data to the server via the internet connection and waits for confirmation of receipt.

[0300] Step 7:

[0301] The server receives the 3D model data and uses AI to convert it into a printable format. The server then checks the quality and output suitability of the converted data.

[0302] Step 8:

[0303] After the server completes the conversion, it sends the data to the 3D printer and begins printing the physical object. The server monitors the printing progress and sends a completion notification to the terminal.

[0304] Step 9:

[0305] The user can receive a notification of printing completion via the terminal and pick up the physical object at a designated location. The user can finally check the object in the real world and use or display it.

[0306] (Example 2)

[0307] Next, Example 2 will be described. In the following description, the data processing device 12 is referred to as a "server", and the smart glasses 214 are referred to as a "terminal".

[0308] It is an issue to provide a system that can flexibly and effectively customize a three-dimensional model based on the user's emotions and improve the user experience. Such a system is required to reflect the user's emotional state in real time and improve emotional and visual satisfaction through the generation of physical objects suitable for that state.

[0309] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0310] In this invention, the server includes means for analyzing sensor information to detect the user's emotions, means for presenting options for three-dimensional models based on the detected emotion information, and means for acquiring the three-dimensional model data selected in the virtual environment. Thereby, it becomes possible to propose and generate a three-dimensional model along with the user's emotional state.

[0311] "Detecting emotions" means analyzing the user's expression and voice to grasp the real-time psychological state.

[0312] "Analyzing sensor information" is a process of estimating the user's physiological and psychological state using data obtained from input devices such as cameras and microphones.

[0313] A "three-dimensional model" is three-dimensional object data generated on a computer that possesses a shape that can be physically output.

[0314] "Presenting options" means showing the user a list of three-dimensional models that appear to be the most suitable based on the user's emotional information.

[0315] A "three-dimensional printing device" is a device that creates physical three-dimensional objects based on digital data.

[0316] "Artificial intelligence" refers to all methods and technologies for performing intelligent tasks using computer programs, and in this invention, it is specifically used for data analysis and generation.

[0317] This invention is a system that recognizes a user's emotions in a virtual environment and recommends and outputs an optimal three-dimensional model based on those emotions. The system comprises a terminal, a server, and a three-dimensional printing device.

[0318] The device uses input devices such as cameras and microphones to capture the user's facial expressions and voice in real time in order to detect the user's emotions. The acquired data is analyzed by an emotion analysis engine to identify the user's physiological and psychological state. Emotion analysis software is used for this analysis.

[0319] The server generates a selection of the most suitable 3D models for the user based on the analysis results from the emotion analysis engine. At this time, it utilizes the generated AI model to extract highly relevant 3D models from the database and presents them to the user.

[0320] The user selects their preferred 3D model from the options provided on the device. This selection information is sent from the device to the server. The generating AI model converts the selected 3D model data into the appropriate output format and prepares it for generation as a physical object in a 3D printer.

[0321] For example, if the emotion engine determines that the user is feeling stressed, it will recommend an object with a relaxing effect (for example, a smooth-shaped ornament). When the user selects this object, the terminal transmits the selection information to the server, which converts the data into a format suitable for a printer and outputs it as a physical object. An example of a prompt message would be, "The user is feeling stressed. Please suggest a 3D model with a relaxing effect."

[0322] This system allows users to obtain a three-dimensional model that reflects their emotional state, resulting in improved visual and emotional satisfaction.

[0323] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0324] Step 1:

[0325] The device uses its camera and microphone to capture the user's facial expressions and voice in real time. The input data includes emotional indicators such as facial movements, facial expressions, and voice tone. The device sends this data to an emotion analysis engine.

[0326] Step 2:

[0327] The emotion analysis engine analyzes the data transmitted from the terminal. Specifically, emotion analysis software is used to estimate the user's psychological state (e.g., joy, sadness, stress) from the data. The output of this process is information indicating the estimated emotional state of the user.

[0328] Step 3:

[0329] The server receives emotional state information obtained from the emotion analysis engine as input. Based on this information, it utilizes a generative AI model to generate a selection of 3D models suitable for the emotion. This output is, for example, a list of 3D models with shapes that promote relaxation.

[0330] Step 4:

[0331] The user reviews the 3D model options presented on the device and selects their desired model. The user's selection action is recorded as input on the device, and the selected model information is sent from the device to the server.

[0332] Step 5:

[0333] After receiving the selected model information from the terminal, the server uses a generated AI model to convert the data into an appropriate output format for the 3D printing device. This conversion process includes format conversion and scaling adjustments. The converted data is then obtained as output.

[0334] Step 6:

[0335] The server sends the converted data to the 3D printer. The 3D printer uses this data to begin the process of generating the actual physical object. Specifically, it uses materials to sequentially construct the shape.

[0336] (Application Example 2)

[0337] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0338] Modern consumers demand personalized experiences with goods, but traditional systems struggle to provide this in real time. Furthermore, there's a lack of technology to instantly suggest and physically deliver special products that cater to consumer emotions and preferences. This hinders improvements in customer satisfaction in retail stores.

[0339] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0340] In this invention, the server includes means for analyzing the user's emotional state in a virtual space and recommending a three-dimensional model that is individually suited to the user; means for converting the three-dimensional model data into a specific output format using artificial intelligence; and means for transmitting the converted data to a three-dimensional printing device and outputting it as a physical structure. This makes it possible to provide personalized goods based on the consumer's emotions.

[0341] A "virtual space" is an artificial environment created by a computer, a space in which users can immerse themselves and experience various sensory sensations.

[0342] "User's emotional state" refers to an individual's psychological or emotional condition, which is analyzed from facial expressions, voice, and other factors.

[0343] A "three-dimensional model" is a model of a three-dimensional shape or structure, represented physically or digitally.

[0344] "Artificial intelligence" is a general term for human intelligent behavioral processes developed with the aim of being imitated by computers.

[0345] A "specific output format" refers to a format or style that is suitable for a particular purpose or use.

[0346] A "3D printing device" is a machine that materializes physical objects by constructing them layer by layer based on three-dimensional design data.

[0347] A "physical structure" is a real-world object or shape that can be touched in actual space.

[0348] A "person of interest" refers to an individual or group that shows interest in a particular service or product.

[0349] "Emotion-based recommendations" refer to suggestions or recommendations that reflect an individual's psychological state.

[0350] A "buying space" refers to a physical or digital marketplace where products and services are traded.

[0351] In embodiments of the present invention, the system consists of a user, a terminal, a server, and a 3D printing device. The user accesses a virtual space through a dedicated terminal and engages in activities within it through actions, facial expressions, and voice. The terminal senses the user's facial expressions and voice in real time and analyzes the user's emotional state using an emotion analysis engine.

[0352] At this stage, the terminal presents the user with appropriate 3D model options based on the analyzed emotions. If the user selects a specific 3D model, that information is transferred to the server. The server uses a generative AI model to convert the 3D model data into a specific output format and sends that data to a 3D printing device to output it as a physical structure.

[0353] For example, if the device analyzes that the user is in an emotional state seeking relaxation, it will recommend a shape aimed at stress reduction, such as a rounded object. If the user selects this three-dimensional shape, the model is output by a 3D printer via the server and provided to the user in real time.

[0354] In this process, the generative AI model assists in automatically generating appropriate models by using prompts such as, "If the user's emotion is one of relaxation, what kind of 3D model design should be proposed?"

[0355] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0356] Step 1:

[0357] The device acquires the user's facial expressions and voice data. This data is input collected in real time through the camera and microphone. The data obtained is then analyzed by an emotion analysis engine, and the user's emotional state is output.

[0358] Step 2:

[0359] The device presents the user with appropriate 3D model options based on the output of its emotion analysis engine. If the emotional state is relaxed, a 3D model with a smooth shape is suggested. A generative AI model is used for the suggestions, and the prompt "What 3D model should be presented based on the emotion?" is input, and a list of appropriate models is output.

[0360] Step 3:

[0361] The user selects their preferred 3D model from those presented. This selection is entered into the terminal and transmitted to the server. The user's selection information is sent to the server as input data, and this information is used in the next conversion process.

[0362] Step 4:

[0363] The server converts the user's selected 3D model data into a specific output format. This process again utilizes a generative AI model to adjust the model data into a format suitable for a 3D printer. The input is the user's selected model data, and the output is data usable by the 3D printer.

[0364] Step 5:

[0365] The converted output data is sent from the server to the 3D printing machine. The printing machine receives this data and outputs a model as a physical structure. Once the process from data transmission to model printing is complete, it is finally provided to the user.

[0366] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0367] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0368] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0369] [Third Embodiment]

[0370] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0371] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0372] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0373] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0374] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0375] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0376] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0377] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0378] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0379] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0380] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0381] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0382] This invention provides a system that includes a program for generating a three-dimensional model selected by a user within a virtual environment. The system includes a terminal, a server, and a three-dimensional printing device. The specific processing of each component is described below.

[0383] The user selects a 3D object through a virtual reality interface on their device. This selection causes the device to retrieve the corresponding 3D model data and send it to a server. The server then analyzes the received data and uses artificial intelligence to automatically convert it into a specific output format usable by a 3D printer. The converted data is then sent from the server to the 3D printer and output as a physical object.

[0384] For example, if a user selects a sculpture they are interested in within a virtual exhibition and wishes to replicate its 3D model data in the real world, the terminal captures the selected data and sends it to a server. The server processes, transforms, and optimizes the data, then outputs the physical object using a designated 3D printer. This entire process allows users to easily and quickly bring objects from the virtual environment into the real world.

[0385] This system provides an efficient and user-friendly method for physically recreating any three-dimensional model that a user finds in virtual reality. Users can quickly achieve their goals without special technical knowledge, as automated processing and 3D printing are performed via the server through their terminal.

[0386] The following describes the processing flow.

[0387] Step 1:

[0388] The user selects 3D objects within a virtual environment. The user visually confirms and selects objects of interest through a virtual reality interface.

[0389] Step 2:

[0390] The device captures the selected 3D model data. The device receives the user's selection and temporarily saves the 3D model data in its internal memory.

[0391] Step 3:

[0392] The device sends 3D model data to the server. The device sends the saved data to the server via the internet connection and verifies that it was sent correctly.

[0393] Step 4:

[0394] The server analyzes the received data. The server checks the incoming 3D model data and determines whether the conversion process can proceed without problems.

[0395] Step 5:

[0396] The server uses artificial intelligence to transform the data. The server passes the analyzed data to a generating AI, which then converts it into a specific output format that can be used by a 3D printing device.

[0397] Step 6:

[0398] The server verifies the converted data. The server checks the integrity of the output data and optimizes it to ensure data quality.

[0399] Step 7:

[0400] The server sends the converted and optimized data to the 3D printing device. The server then sends the final data to the designated printing device via the internet.

[0401] Step 8:

[0402] The 3D printing device creates a physical object based on the transmitted data. Upon completion, the user receives a notification that they can receive the physical object.

[0403] (Example 1)

[0404] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0405] Traditional 3D printing processes required advanced technical knowledge and complex procedures to reproduce a 3D model selected by the user in a virtual environment in the real world. Furthermore, data conversion and print optimization were often performed manually, which was time-consuming and labor-intensive, making it difficult to generate objects quickly and intuitively.

[0406] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0407] In this invention, the server includes means for acquiring data of an object model selected in a virtual environment via a terminal operated by the user, means for analyzing the data of the object model and converting it into a specific printable format using a machine learning algorithm, and means for transmitting the converted data to a printing device and materializing it as an object. This makes it possible for the user to quickly and intuitively reproduce a model in the virtual environment physically without requiring any special technical knowledge.

[0408] A "terminal" is an electronic device that a user operates to select objects within a virtual environment and transmit that selection information to a server.

[0409] A "server" is a computer system that receives data sent from a terminal, analyzes and converts it, and then sends it to a printing device.

[0410] An "object model" is digital data of a three-dimensional shape that can be selected by the user within a virtual environment.

[0411] A "machine learning algorithm" is an artificial intelligence computation method used to analyze data and convert it into a specific format.

[0412] A "printing device" is equipment used to generate physical objects based on converted digital data.

[0413] "Conversion" is the process of automatically changing the data of an object model into a printable format.

[0414] "Optimization" refers to adjusting data and processes to perform printing efficiently.

[0415] "Manifestation" refers to the process of outputting model data from a virtual environment in a physical form.

[0416] This invention is a system comprising a user-operated terminal, a server for analyzing and converting data, and a printing device for generating physical objects. The user selects a three-dimensional object of interest using a virtual reality interface on the terminal. The terminal acquires the digital data of the selected three-dimensional model and transmits that data to the server. The server utilizes a generated AI model to analyze and convert the model data, automatically converting it into a format usable by the printing device (e.g., G-code).

[0417] For example, if a user wants to recreate a specific sculpture they saw in a virtual museum, they select the sculpture on their device. The device then captures digital data and sends it to a server. The server uses a generative AI model to convert the data into the appropriate format and sends it to a printing device, thereby generating the physical sculpture.

[0418] In this invention, an example of a generated prompt message might be an instruction such as, "Please convert the selected three-dimensional model into a printable format." This system helps users easily reproduce a model selected in a virtual environment as a real object, even without special technical knowledge.

[0419] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0420] Step 1:

[0421] The user selects a 3D object of interest using the virtual reality interface on the terminal. The input is the object ID selected by the user in the virtual space. Based on this selection, the terminal retrieves the corresponding 3D model data and sends it to the server. The output is the 3D model data sent to the server.

[0422] Step 2:

[0423] The server receives 3D model data sent from the terminal. The input is the 3D model data sent from the terminal. The server analyzes the data using a generative AI model and performs data processing to convert it into a specific printable format (e.g., G-code). The output is the data converted into a format usable by a printing device.

[0424] Step 3:

[0425] The server sends the converted data to the printer. The input is the printable data converted by the server. The server sends this data to the printer and performs actions to instruct it to materialize an object. The output is the physical object produced by the printer.

[0426] This process allows users to quickly and intuitively recreate selected three-dimensional objects in the real world.

[0427] (Application Example 1)

[0428] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0429] There is a need to efficiently reproduce three-dimensional representations selected in a virtual space using visual input devices as physical objects in the real world, by modifying them to the shape desired by the user. Conventional methods often require advanced expertise and complex operations to convert data selected and modified by the user in the virtual space into physical objects, making them difficult for the average user.

[0430] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0431] In this invention, the server includes means for acquiring three-dimensional representation data selected in a virtual space, means for converting the three-dimensional representation data into a specific output format using a machine learning model that generates the data, means for transmitting the converted data to a manufacturing device and outputting it as a physical item, and means for acquiring and reflecting selection and modification information of items in the virtual space using a visual input device. This makes it possible for users to efficiently and quickly materialize items they have selected and modified in the real world from the virtual space without requiring specialized knowledge.

[0432] A "virtual space" is a computer-generated environment that users can visually experience through digital technology.

[0433] "Three-dimensional representation data" refers to digital data that contains shape information constructed in three dimensions, and is a concrete representation of the shape that is visually experienced in a virtual space.

[0434] A "machine learning model" is a type of algorithm that learns from large amounts of data and transforms or recognizes input data to suit a specific purpose.

[0435] An "output format" is a standard for representing data obtained through machine processing in a physically or digitally defined format.

[0436] "Manufacturing equipment" refers to devices used to create physical objects or structures based on digital data, and three-dimensional printers are a prime example of this.

[0437] A "visual input device" is a device that acquires a user's visual information and processes it digitally, and includes, for example, smart glasses.

[0438] This invention provides a system and process for materializing objects selected in a virtual space as physical shapes in the real world. Specific embodiments thereof are described below.

[0439] The user uses smart glasses, a visual input device, to select any 3D representation in a virtual space. The user can also modify the color, size, and material of that 3D representation. This information is transmitted to the terminal in real time via edge computing.

[0440] The terminal sends the acquired 3D representation data to a cloud server. This server is equipped with a machine learning model using the TensorFlow library, which analyzes the transmitted data and converts it into an output format usable by the manufacturing equipment. The conversion results are then sent to the manufacturing equipment.

[0441] The manufacturing equipment includes 3D printers that produce physical objects based on digital data. These 3D printers quickly and accurately output objects selected and modified by the user in a virtual space based on the received data.

[0442] As a concrete example, consider a scenario where a user selects a blue chair in a virtual store and adjusts its size. The 3D representation data selected through smart glasses is quickly processed on a cloud server and output by a designated 3D printer. This allows the user to receive a real-world object that directly reflects their visual selection in the virtual space.

[0443] An example of a prompt for a generated AI model is, "The selected 3D representation is a blue chair. Adjust the size and convert it into data for printing on a real-world 3D printer." This prompt serves to instruct the server to perform the optimal data conversion.

[0444] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0445] Step 1:

[0446] The user uses smart glasses, a visual input device, to select a 3D representation within a virtual space. Sensors in the glasses detect the user's gaze and gesture input, and transmit this selection information to the terminal via edge computing. The input at this time is the selected 3D representation data, and the output is the transmission of that data.

[0447] Step 2:

[0448] The terminal temporarily stores the received 3D representation data and then sends it to the cloud server. This data transfer, based on user input, is performed quickly and securely. Furthermore, data checks are performed using a communication protocol to ensure error-free transmission. The input is the stored 3D representation data, and the output is sending this data to the cloud server.

[0449] Step 3:

[0450] The server runs a machine learning model using the TensorFlow library on the cloud and analyzes the received 3D representation data. The analysis results are converted into a specific output format usable by 3D printers. The input is the transmitted 3D representation data, and the output is the data converted into the specific output format. The analysis also includes data correction and noise reduction using a generative AI model.

[0451] Step 4:

[0452] The server sends the converted data to the manufacturing equipment, specifically a 3D printer. This transmission uses network communication, and the server also checks the status of the receiving manufacturing equipment (running, stopped, etc.). The input is the converted output format data, and the output is the data transmission to the 3D printer.

[0453] Step 5:

[0454] The 3D printer, a manufacturing device, begins printing a physical object based on the received data. The printer is calibrated, and the material selection and layer configuration of the object to be printed are optimized. The input is data sent from the server, and the output is the completed physical object.

[0455] This processing flow allows users to materialize selected objects in the virtual world as physical objects in the real world. Because this series of operations is performed quickly, users can experience a seamless transformation from digital to physical.

[0456] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0457] This invention relates to a system that recognizes a user's emotions in a virtual environment and recommends and outputs an optimal three-dimensional model based on those emotions. The system includes a terminal, a server, a three-dimensional printing device, and an emotion engine that analyzes the user's emotions.

[0458] When a user engages in activities within the virtual environment, the device has a function that senses the user's facial expressions and voice input, and based on this, an emotion engine analyzes the user's emotional state. The emotion engine determines the user's emotions in real time and provides a selection of 3D models that match those emotions.

[0459] When a user accepts an emotion-based recommendation and selects a specific 3D object, the terminal confirms the selection and transfers the 3D model data to the server. The server has the capability to analyze the data and convert it into a format suitable for a 3D printer using artificial intelligence. After conversion, the server sends the data to the 3D printer and executes the process of outputting it as a physical object.

[0460] For example, if the emotion engine determines that a user is feeling stressed, it can recommend objects with a relaxing effect, such as a smooth-shaped ornament. If the user selects this, it will materialize in the real world as a design that promotes relaxation.

[0461] By incorporating an emotional engine, the system can provide flexible services that adapt to the user's current mental and physical state. This allows users to go beyond a mere visual experience and enhance their emotional satisfaction.

[0462] The following describes the processing flow.

[0463] Step 1:

[0464] The user logs into the virtual environment and begins their activities. The user interacts with the interface to explore multiple visually presented three-dimensional objects.

[0465] Step 2:

[0466] The device collects emotional data from the user's facial expressions and voice. The device uses built-in sensors to monitor the user's facial movements and voice tone in real time and transmits the data to the emotion engine.

[0467] Step 3:

[0468] The emotion engine analyzes the received data to identify the user's current emotional state. The emotion engine uses algorithms to determine the user's emotions, such as stress, joy, and surprise.

[0469] Step 4:

[0470] The emotion engine generates a list of recommended 3D objects based on the user's emotions and sends it to the terminal. The terminal then presents this list to the user via a control panel or overlay display within the virtual environment.

[0471] Step 5:

[0472] The user selects a 3D object of interest from emotion-based recommendations. The user confirms their selection by pointing to the object or pressing a select button.

[0473] Step 6:

[0474] The terminal sends the selected 3D model data to the server. The terminal securely uploads the data to the server via the internet connection and waits for confirmation of receipt.

[0475] Step 7:

[0476] The server receives the 3D model data and uses AI to convert it into a printable format. The server then checks the quality and output suitability of the converted data.

[0477] Step 8:

[0478] After the server completes the conversion, it sends the data to the 3D printer and begins printing the physical object. The server monitors the printing progress and sends a completion notification to the terminal.

[0479] Step 9:

[0480] Users receive notification via their device when printing is complete and can pick up the physical object at a designated location. Ultimately, users can view the object in the real world and use or display it.

[0481] (Example 2)

[0482] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0483] The challenge lies in providing a system that can enhance the user experience by offering flexibly and effectively customized 3D models based on user emotions. Such a system is required to reflect the user's emotional state in real time and improve emotional and visual satisfaction by generating physical objects appropriate to that state.

[0484] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0485] In this invention, the server includes means for analyzing sensor information to detect the user's emotions, means for presenting a selection of three-dimensional models based on the detected emotion information, and means for acquiring the selected three-dimensional model data in a virtual environment. This makes it possible to propose and generate three-dimensional models that are in line with the user's emotional state.

[0486] "Emotion detection" means analyzing the user's facial expressions and voice to understand their psychological state in real time.

[0487] "Analyzing sensor information" is the process of estimating a user's physiological and psychological state using data acquired from input devices such as cameras and microphones.

[0488] A "three-dimensional model" is three-dimensional object data generated on a computer that possesses a shape that can be physically output.

[0489] "Presenting options" means showing the user a list of three-dimensional models that appear to be the most suitable based on the user's emotional information.

[0490] A "three-dimensional printing device" is a device that creates physical three-dimensional objects based on digital data.

[0491] "Artificial intelligence" refers to all methods and technologies for performing intelligent tasks using computer programs, and in this invention, it is specifically used for data analysis and generation.

[0492] This invention is a system that recognizes a user's emotions in a virtual environment and recommends and outputs an optimal three-dimensional model based on those emotions. The system comprises a terminal, a server, and a three-dimensional printing device.

[0493] The device uses input devices such as cameras and microphones to capture the user's facial expressions and voice in real time in order to detect the user's emotions. The acquired data is analyzed by an emotion analysis engine to identify the user's physiological and psychological state. Emotion analysis software is used for this analysis.

[0494] The server generates a selection of the most suitable 3D models for the user based on the analysis results from the emotion analysis engine. At this time, it utilizes the generated AI model to extract highly relevant 3D models from the database and presents them to the user.

[0495] The user selects their preferred 3D model from the options provided on the device. This selection information is sent from the device to the server. The generating AI model converts the selected 3D model data into the appropriate output format and prepares it for generation as a physical object in a 3D printer.

[0496] For example, if the emotion engine determines that the user is feeling stressed, it will recommend an object with a relaxing effect (for example, a smooth-shaped ornament). When the user selects this object, the terminal transmits the selection information to the server, which converts the data into a format suitable for a printer and outputs it as a physical object. An example of a prompt message would be, "The user is feeling stressed. Please suggest a 3D model with a relaxing effect."

[0497] This system allows users to obtain a three-dimensional model that reflects their emotional state, resulting in improved visual and emotional satisfaction.

[0498] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0499] Step 1:

[0500] The device uses its camera and microphone to capture the user's facial expressions and voice in real time. The input data includes emotional indicators such as facial movements, facial expressions, and voice tone. The device sends this data to an emotion analysis engine.

[0501] Step 2:

[0502] The emotion analysis engine analyzes the data transmitted from the terminal. Specifically, emotion analysis software is used to estimate the user's psychological state (e.g., joy, sadness, stress) from the data. The output of this process is information indicating the estimated emotional state of the user.

[0503] Step 3:

[0504] The server receives emotional state information obtained from the emotion analysis engine as input. Based on this information, it utilizes a generative AI model to generate a selection of 3D models suitable for the emotion. This output is, for example, a list of 3D models with shapes that promote relaxation.

[0505] Step 4:

[0506] The user reviews the 3D model options presented on the device and selects their desired model. The user's selection action is recorded as input on the device, and the selected model information is sent from the device to the server.

[0507] Step 5:

[0508] After receiving the selected model information from the terminal, the server uses a generated AI model to convert the data into an appropriate output format for the 3D printing device. This conversion process includes format conversion and scaling adjustments. The converted data is then obtained as output.

[0509] Step 6:

[0510] The server sends the converted data to the 3D printer. The 3D printer uses this data to begin the process of generating the actual physical object. Specifically, it uses materials to sequentially construct the shape.

[0511] (Application Example 2)

[0512] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0513] Modern consumers demand personalized experiences with goods, but traditional systems struggle to provide this in real time. Furthermore, there's a lack of technology to instantly suggest and physically deliver special products that cater to consumer emotions and preferences. This hinders improvements in customer satisfaction in retail stores.

[0514] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0515] In this invention, the server includes means for analyzing the user's emotional state in a virtual space and recommending a three-dimensional model that is individually suited to the user; means for converting the three-dimensional model data into a specific output format using artificial intelligence; and means for transmitting the converted data to a three-dimensional printing device and outputting it as a physical structure. This makes it possible to provide personalized goods based on the consumer's emotions.

[0516] A "virtual space" is an artificial environment created by a computer, a space in which users can immerse themselves and experience various sensory sensations.

[0517] "User's emotional state" refers to an individual's psychological or emotional condition, which is analyzed from facial expressions, voice, and other factors.

[0518] A "three-dimensional model" is a model of a three-dimensional shape or structure, represented physically or digitally.

[0519] "Artificial intelligence" is a general term for human intelligent behavioral processes developed with the aim of being imitated by computers.

[0520] A "specific output format" refers to a format or style that is suitable for a particular purpose or use.

[0521] A "3D printing device" is a machine that materializes physical objects by constructing them layer by layer based on three-dimensional design data.

[0522] A "physical structure" is a real-world object or shape that can be touched in actual space.

[0523] A "person of interest" refers to an individual or group that shows interest in a particular service or product.

[0524] "Emotion-based recommendations" refer to suggestions or recommendations that reflect an individual's psychological state.

[0525] A "buying space" refers to a physical or digital marketplace where products and services are traded.

[0526] In embodiments of the present invention, the system consists of a user, a terminal, a server, and a 3D printing device. The user accesses a virtual space through a dedicated terminal and engages in activities within it through actions, facial expressions, and voice. The terminal senses the user's facial expressions and voice in real time and analyzes the user's emotional state using an emotion analysis engine.

[0527] At this stage, the terminal presents the user with appropriate 3D model options based on the analyzed emotions. If the user selects a specific 3D model, that information is transferred to the server. The server uses a generative AI model to convert the 3D model data into a specific output format and sends that data to a 3D printing device to output it as a physical structure.

[0528] For example, if the device analyzes that the user is in an emotional state seeking relaxation, it will recommend a shape aimed at stress reduction, such as a rounded object. If the user selects this three-dimensional shape, the model is output by a 3D printer via the server and provided to the user in real time.

[0529] In this process, the generative AI model assists in automatically generating appropriate models by using prompts such as, "If the user's emotion is one of relaxation, what kind of 3D model design should be proposed?"

[0530] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0531] Step 1:

[0532] The device acquires the user's facial expressions and voice data. This data is input collected in real time through the camera and microphone. The data obtained is then analyzed by an emotion analysis engine, and the user's emotional state is output.

[0533] Step 2:

[0534] The device presents the user with appropriate 3D model options based on the output of its emotion analysis engine. If the emotional state is relaxed, a 3D model with a smooth shape is suggested. A generative AI model is used for the suggestions, and the prompt "What 3D model should be presented based on the emotion?" is input, and a list of appropriate models is output.

[0535] Step 3:

[0536] The user selects their preferred 3D model from those presented. This selection is entered into the terminal and transmitted to the server. The user's selection information is sent to the server as input data, and this information is used in the next conversion process.

[0537] Step 4:

[0538] The server converts the user's selected 3D model data into a specific output format. This process again utilizes a generative AI model to adjust the model data into a format suitable for a 3D printer. The input is the user's selected model data, and the output is data usable by the 3D printer.

[0539] Step 5:

[0540] The converted output data is sent from the server to the 3D printing machine. The printing machine receives this data and outputs a model as a physical structure. Once the process from data transmission to model printing is complete, it is finally provided to the user.

[0541] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0542] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0543] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0544] [Fourth Embodiment]

[0545] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0546] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0547] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0548] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0549] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0550] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0551] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0552] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0553] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0554] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0555] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0556] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0557] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0558] This invention provides a system that includes a program for generating a three-dimensional model selected by a user within a virtual environment. The system includes a terminal, a server, and a three-dimensional printing device. The specific processing of each component is described below.

[0559] The user selects a 3D object through a virtual reality interface on their device. This selection causes the device to retrieve the corresponding 3D model data and send it to a server. The server then analyzes the received data and uses artificial intelligence to automatically convert it into a specific output format usable by a 3D printer. The converted data is then sent from the server to the 3D printer and output as a physical object.

[0560] For example, if a user selects a sculpture they are interested in within a virtual exhibition and wishes to replicate its 3D model data in the real world, the terminal captures the selected data and sends it to a server. The server processes, transforms, and optimizes the data, then outputs the physical object using a designated 3D printer. This entire process allows users to easily and quickly bring objects from the virtual environment into the real world.

[0561] This system provides an efficient and user-friendly method for physically recreating any three-dimensional model that a user finds in virtual reality. Users can quickly achieve their goals without special technical knowledge, as automated processing and 3D printing are performed via the server through their terminal.

[0562] The following describes the processing flow.

[0563] Step 1:

[0564] The user selects 3D objects within a virtual environment. The user visually confirms and selects objects of interest through a virtual reality interface.

[0565] Step 2:

[0566] The device captures the selected 3D model data. The device receives the user's selection and temporarily saves the 3D model data in its internal memory.

[0567] Step 3:

[0568] The device sends 3D model data to the server. The device sends the saved data to the server via the internet connection and verifies that it was sent correctly.

[0569] Step 4:

[0570] The server analyzes the received data. The server checks the incoming 3D model data and determines whether the conversion process can proceed without problems.

[0571] Step 5:

[0572] The server uses artificial intelligence to transform the data. The server passes the analyzed data to a generating AI, which then converts it into a specific output format that can be used by a 3D printing device.

[0573] Step 6:

[0574] The server verifies the converted data. The server checks the integrity of the output data and optimizes it to ensure data quality.

[0575] Step 7:

[0576] The server sends the converted and optimized data to the 3D printing device. The server then sends the final data to the designated printing device via the internet.

[0577] Step 8:

[0578] The 3D printing device creates a physical object based on the transmitted data. Upon completion, the user receives a notification that they can receive the physical object.

[0579] (Example 1)

[0580] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0581] Traditional 3D printing processes required advanced technical knowledge and complex procedures to reproduce a 3D model selected by the user in a virtual environment in the real world. Furthermore, data conversion and print optimization were often performed manually, which was time-consuming and labor-intensive, making it difficult to generate objects quickly and intuitively.

[0582] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0583] In this invention, the server includes means for acquiring data of an object model selected in a virtual environment via a terminal operated by the user, means for analyzing the data of the object model and converting it into a specific printable format using a machine learning algorithm, and means for transmitting the converted data to a printing device and materializing it as an object. This makes it possible for the user to quickly and intuitively reproduce a model in the virtual environment physically without requiring any special technical knowledge.

[0584] A "terminal" is an electronic device that a user operates to select objects within a virtual environment and transmit that selection information to a server.

[0585] A "server" is a computer system that receives data sent from a terminal, analyzes and converts it, and then sends it to a printing device.

[0586] An "object model" is digital data of a three-dimensional shape that can be selected by the user within a virtual environment.

[0587] A "machine learning algorithm" is an artificial intelligence computation method used to analyze data and convert it into a specific format.

[0588] A "printing device" is equipment used to generate physical objects based on converted digital data.

[0589] "Conversion" is the process of automatically changing the data of an object model into a printable format.

[0590] "Optimization" refers to adjusting data and processes to perform printing efficiently.

[0591] "Manifestation" refers to the process of outputting model data from a virtual environment in a physical form.

[0592] This invention is a system comprising a user-operated terminal, a server for analyzing and converting data, and a printing device for generating physical objects. The user selects a three-dimensional object of interest using a virtual reality interface on the terminal. The terminal acquires the digital data of the selected three-dimensional model and transmits that data to the server. The server utilizes a generated AI model to analyze and convert the model data, automatically converting it into a format usable by the printing device (e.g., G-code).

[0593] For example, if a user wants to recreate a specific sculpture they saw in a virtual museum, they select the sculpture on their device. The device then captures digital data and sends it to a server. The server uses a generative AI model to convert the data into the appropriate format and sends it to a printing device, thereby generating the physical sculpture.

[0594] In this invention, an example of a generated prompt message might be an instruction such as, "Please convert the selected three-dimensional model into a printable format." This system helps users easily reproduce a model selected in a virtual environment as a real object, even without special technical knowledge.

[0595] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0596] Step 1:

[0597] The user selects a 3D object of interest using the virtual reality interface on the terminal. The input is the object ID selected by the user in the virtual space. Based on this selection, the terminal retrieves the corresponding 3D model data and sends it to the server. The output is the 3D model data sent to the server.

[0598] Step 2:

[0599] The server receives 3D model data sent from the terminal. The input is the 3D model data sent from the terminal. The server analyzes the data using a generative AI model and performs data processing to convert it into a specific printable format (e.g., G-code). The output is the data converted into a format usable by a printing device.

[0600] Step 3:

[0601] The server sends the converted data to the printer. The input is the printable data converted by the server. The server sends this data to the printer and performs actions to instruct it to materialize an object. The output is the physical object produced by the printer.

[0602] This process allows users to quickly and intuitively recreate selected three-dimensional objects in the real world.

[0603] (Application Example 1)

[0604] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0605] There is a need to efficiently reproduce three-dimensional representations selected in a virtual space using visual input devices as physical objects in the real world, by modifying them to the shape desired by the user. Conventional methods often require advanced expertise and complex operations to convert data selected and modified by the user in the virtual space into physical objects, making them difficult for the average user.

[0606] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0607] In this invention, the server includes means for acquiring three-dimensional representation data selected in a virtual space, means for converting the three-dimensional representation data into a specific output format using a machine learning model that generates the data, means for transmitting the converted data to a manufacturing device and outputting it as a physical item, and means for acquiring and reflecting selection and modification information of items in the virtual space using a visual input device. This makes it possible for users to efficiently and quickly materialize items they have selected and modified in the real world from the virtual space without requiring specialized knowledge.

[0608] A "virtual space" is a computer-generated environment that users can visually experience through digital technology.

[0609] "Three-dimensional representation data" refers to digital data that contains shape information constructed in three dimensions, and is a concrete representation of the shape that is visually experienced in a virtual space.

[0610] A "machine learning model" is a type of algorithm that learns from large amounts of data and transforms or recognizes input data to suit a specific purpose.

[0611] An "output format" is a standard for representing data obtained through machine processing in a physically or digitally defined format.

[0612] "Manufacturing equipment" refers to devices used to create physical objects or structures based on digital data, and three-dimensional printers are a prime example of this.

[0613] A "visual input device" is a device that acquires a user's visual information and processes it digitally, and includes, for example, smart glasses.

[0614] This invention provides a system and process for materializing objects selected in a virtual space as physical shapes in the real world. Specific embodiments thereof are described below.

[0615] The user uses smart glasses, a visual input device, to select any 3D representation in a virtual space. The user can also modify the color, size, and material of that 3D representation. This information is transmitted to the terminal in real time via edge computing.

[0616] The terminal sends the acquired 3D representation data to a cloud server. This server is equipped with a machine learning model using the TensorFlow library, which analyzes the transmitted data and converts it into an output format usable by the manufacturing equipment. The conversion results are then sent to the manufacturing equipment.

[0617] The manufacturing equipment includes 3D printers that produce physical objects based on digital data. These 3D printers quickly and accurately output objects selected and modified by the user in a virtual space based on the received data.

[0618] As a concrete example, consider a scenario where a user selects a blue chair in a virtual store and adjusts its size. The 3D representation data selected through smart glasses is quickly processed on a cloud server and output by a designated 3D printer. This allows the user to receive a real-world object that directly reflects their visual selection in the virtual space.

[0619] An example of a prompt for a generated AI model is, "The selected 3D representation is a blue chair. Adjust the size and convert it into data for printing on a real-world 3D printer." This prompt serves to instruct the server to perform the optimal data conversion.

[0620] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0621] Step 1:

[0622] The user uses smart glasses, a visual input device, to select a 3D representation within a virtual space. Sensors in the glasses detect the user's gaze and gesture input, and transmit this selection information to the terminal via edge computing. The input at this time is the selected 3D representation data, and the output is the transmission of that data.

[0623] Step 2:

[0624] The terminal temporarily stores the received 3D representation data and then sends it to the cloud server. This data transfer, based on user input, is performed quickly and securely. Furthermore, data checks are performed using a communication protocol to ensure error-free transmission. The input is the stored 3D representation data, and the output is sending this data to the cloud server.

[0625] Step 3:

[0626] The server runs a machine learning model using the TensorFlow library on the cloud and analyzes the received 3D representation data. The analysis results are converted into a specific output format usable by 3D printers. The input is the transmitted 3D representation data, and the output is the data converted into the specific output format. The analysis also includes data correction and noise reduction using a generative AI model.

[0627] Step 4:

[0628] The server sends the converted data to the manufacturing equipment, specifically a 3D printer. This transmission uses network communication, and the server also checks the status of the receiving manufacturing equipment (running, stopped, etc.). The input is the converted output format data, and the output is the data transmission to the 3D printer.

[0629] Step 5:

[0630] The 3D printer, a manufacturing device, begins printing a physical object based on the received data. The printer is calibrated, and the material selection and layer configuration of the object to be printed are optimized. The input is data sent from the server, and the output is the completed physical object.

[0631] This processing flow allows users to materialize selected objects in the virtual world as physical objects in the real world. Because this series of operations is performed quickly, users can experience a seamless transformation from digital to physical.

[0632] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0633] This invention relates to a system that recognizes a user's emotions in a virtual environment and recommends and outputs an optimal three-dimensional model based on those emotions. The system includes a terminal, a server, a three-dimensional printing device, and an emotion engine that analyzes the user's emotions.

[0634] When a user engages in activities within the virtual environment, the device has a function that senses the user's facial expressions and voice input, and based on this, an emotion engine analyzes the user's emotional state. The emotion engine determines the user's emotions in real time and provides a selection of 3D models that match those emotions.

[0635] When a user accepts an emotion-based recommendation and selects a specific 3D object, the terminal confirms the selection and transfers the 3D model data to the server. The server has the capability to analyze the data and convert it into a format suitable for a 3D printer using artificial intelligence. After conversion, the server sends the data to the 3D printer and executes the process of outputting it as a physical object.

[0636] For example, if the emotion engine determines that a user is feeling stressed, it can recommend objects with a relaxing effect, such as a smooth-shaped ornament. If the user selects this, it will materialize in the real world as a design that promotes relaxation.

[0637] By incorporating an emotional engine, the system can provide flexible services that adapt to the user's current mental and physical state. This allows users to go beyond a mere visual experience and enhance their emotional satisfaction.

[0638] The following describes the processing flow.

[0639] Step 1:

[0640] The user logs into the virtual environment and begins their activities. The user interacts with the interface to explore multiple visually presented three-dimensional objects.

[0641] Step 2:

[0642] The device collects emotional data from the user's facial expressions and voice. The device uses built-in sensors to monitor the user's facial movements and voice tone in real time and transmits the data to the emotion engine.

[0643] Step 3:

[0644] The emotion engine analyzes the received data to identify the user's current emotional state. The emotion engine uses algorithms to determine the user's emotions, such as stress, joy, and surprise.

[0645] Step 4:

[0646] The emotion engine generates a list of recommended 3D objects based on the user's emotions and sends it to the terminal. The terminal then presents this list to the user via a control panel or overlay display within the virtual environment.

[0647] Step 5:

[0648] The user selects a 3D object of interest from emotion-based recommendations. The user confirms their selection by pointing to the object or pressing a select button.

[0649] Step 6:

[0650] The terminal sends the selected 3D model data to the server. The terminal securely uploads the data to the server via the internet connection and waits for confirmation of receipt.

[0651] Step 7:

[0652] The server receives the 3D model data and uses AI to convert it into a printable format. The server then checks the quality and output suitability of the converted data.

[0653] Step 8:

[0654] After the server completes the conversion, it sends the data to the 3D printer and begins printing the physical object. The server monitors the printing progress and sends a completion notification to the terminal.

[0655] Step 9:

[0656] Users receive notification via their device when printing is complete and can pick up the physical object at a designated location. Ultimately, users can view the object in the real world and use or display it.

[0657] (Example 2)

[0658] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0659] The challenge lies in providing a system that can enhance the user experience by offering flexibly and effectively customized 3D models based on user emotions. Such a system is required to reflect the user's emotional state in real time and improve emotional and visual satisfaction by generating physical objects appropriate to that state.

[0660] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0661] In this invention, the server includes means for analyzing sensor information to detect the user's emotions, means for presenting a selection of three-dimensional models based on the detected emotion information, and means for acquiring the selected three-dimensional model data in a virtual environment. This makes it possible to propose and generate three-dimensional models that are in line with the user's emotional state.

[0662] "Emotion detection" means analyzing the user's facial expressions and voice to understand their psychological state in real time.

[0663] "Analyzing sensor information" is the process of estimating a user's physiological and psychological state using data acquired from input devices such as cameras and microphones.

[0664] A "three-dimensional model" is three-dimensional object data generated on a computer that possesses a shape that can be physically output.

[0665] "Presenting options" means showing the user a list of three-dimensional models that appear to be the most suitable based on the user's emotional information.

[0666] A "three-dimensional printing device" is a device that creates physical three-dimensional objects based on digital data.

[0667] "Artificial intelligence" refers to all methods and technologies for performing intelligent tasks using computer programs, and in this invention, it is specifically used for data analysis and generation.

[0668] This invention is a system that recognizes a user's emotions in a virtual environment and recommends and outputs an optimal three-dimensional model based on those emotions. The system comprises a terminal, a server, and a three-dimensional printing device.

[0669] The device uses input devices such as cameras and microphones to capture the user's facial expressions and voice in real time in order to detect the user's emotions. The acquired data is analyzed by an emotion analysis engine to identify the user's physiological and psychological state. Emotion analysis software is used for this analysis.

[0670] The server generates a selection of the most suitable 3D models for the user based on the analysis results from the emotion analysis engine. At this time, it utilizes the generated AI model to extract highly relevant 3D models from the database and presents them to the user.

[0671] The user selects their preferred 3D model from the options provided on the device. This selection information is sent from the device to the server. The generating AI model converts the selected 3D model data into the appropriate output format and prepares it for generation as a physical object in a 3D printer.

[0672] For example, if the emotion engine determines that the user is feeling stressed, it will recommend an object with a relaxing effect (for example, a smooth-shaped ornament). When the user selects this object, the terminal transmits the selection information to the server, which converts the data into a format suitable for a printer and outputs it as a physical object. An example of a prompt message would be, "The user is feeling stressed. Please suggest a 3D model with a relaxing effect."

[0673] This system allows users to obtain a three-dimensional model that reflects their emotional state, resulting in improved visual and emotional satisfaction.

[0674] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0675] Step 1:

[0676] The device uses its camera and microphone to capture the user's facial expressions and voice in real time. The input data includes emotional indicators such as facial movements, facial expressions, and voice tone. The device sends this data to an emotion analysis engine.

[0677] Step 2:

[0678] The emotion analysis engine analyzes the data transmitted from the terminal. Specifically, emotion analysis software is used to estimate the user's psychological state (e.g., joy, sadness, stress) from the data. The output of this process is information indicating the estimated emotional state of the user.

[0679] Step 3:

[0680] The server receives emotional state information obtained from the emotion analysis engine as input. Based on this information, it utilizes a generative AI model to generate a selection of 3D models suitable for the emotion. This output is, for example, a list of 3D models with shapes that promote relaxation.

[0681] Step 4:

[0682] The user reviews the 3D model options presented on the device and selects their desired model. The user's selection action is recorded as input on the device, and the selected model information is sent from the device to the server.

[0683] Step 5:

[0684] After receiving the selected model information from the terminal, the server uses a generated AI model to convert the data into an appropriate output format for the 3D printing device. This conversion process includes format conversion and scaling adjustments. The converted data is then obtained as output.

[0685] Step 6:

[0686] The server sends the converted data to the 3D printer. The 3D printer uses this data to begin the process of generating the actual physical object. Specifically, it uses materials to sequentially construct the shape.

[0687] (Application Example 2)

[0688] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0689] Modern consumers demand personalized experiences with goods, but traditional systems struggle to provide this in real time. Furthermore, there's a lack of technology to instantly suggest and physically deliver special products that cater to consumer emotions and preferences. This hinders improvements in customer satisfaction in retail stores.

[0690] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0691] In this invention, the server includes means for analyzing the user's emotional state in a virtual space and recommending a three-dimensional model that is individually suited to the user; means for converting the three-dimensional model data into a specific output format using artificial intelligence; and means for transmitting the converted data to a three-dimensional printing device and outputting it as a physical structure. This makes it possible to provide personalized goods based on the consumer's emotions.

[0692] A "virtual space" is an artificial environment created by a computer, a space in which users can immerse themselves and experience various sensory sensations.

[0693] "User's emotional state" refers to an individual's psychological or emotional condition, which is analyzed from facial expressions, voice, and other factors.

[0694] A "three-dimensional model" is a model of a three-dimensional shape or structure, represented physically or digitally.

[0695] "Artificial intelligence" is a general term for human intelligent behavioral processes developed with the aim of being imitated by computers.

[0696] A "specific output format" refers to a format or style that is suitable for a particular purpose or use.

[0697] A "3D printing device" is a machine that materializes physical objects by constructing them layer by layer based on three-dimensional design data.

[0698] A "physical structure" is a real-world object or shape that can be touched in actual space.

[0699] A "person of interest" refers to an individual or group that shows interest in a particular service or product.

[0700] "Emotion-based recommendations" refer to suggestions or recommendations that reflect an individual's psychological state.

[0701] A "buying space" refers to a physical or digital marketplace where products and services are traded.

[0702] In embodiments of the present invention, the system consists of a user, a terminal, a server, and a 3D printing device. The user accesses a virtual space through a dedicated terminal and engages in activities within it through actions, facial expressions, and voice. The terminal senses the user's facial expressions and voice in real time and analyzes the user's emotional state using an emotion analysis engine.

[0703] At this stage, the terminal presents the user with appropriate 3D model options based on the analyzed emotions. If the user selects a specific 3D model, that information is transferred to the server. The server uses a generative AI model to convert the 3D model data into a specific output format and sends that data to a 3D printing device to output it as a physical structure.

[0704] For example, if the device analyzes that the user is in an emotional state seeking relaxation, it will recommend a shape aimed at stress reduction, such as a rounded object. If the user selects this three-dimensional shape, the model is output by a 3D printer via the server and provided to the user in real time.

[0705] In this process, the generative AI model assists in automatically generating appropriate models by using prompts such as, "If the user's emotion is one of relaxation, what kind of 3D model design should be proposed?"

[0706] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0707] Step 1:

[0708] The device acquires the user's facial expressions and voice data. This data is input collected in real time through the camera and microphone. The data obtained is then analyzed by an emotion analysis engine, and the user's emotional state is output.

[0709] Step 2:

[0710] The device presents the user with appropriate 3D model options based on the output of its emotion analysis engine. If the emotional state is relaxed, a 3D model with a smooth shape is suggested. A generative AI model is used for the suggestions, and the prompt "What 3D model should be presented based on the emotion?" is input, and a list of appropriate models is output.

[0711] Step 3:

[0712] The user selects their preferred 3D model from those presented. This selection is entered into the terminal and transmitted to the server. The user's selection information is sent to the server as input data, and this information is used in the next conversion process.

[0713] Step 4:

[0714] The server converts the user's selected 3D model data into a specific output format. This process again utilizes a generative AI model to adjust the model data into a format suitable for a 3D printer. The input is the user's selected model data, and the output is data usable by the 3D printer.

[0715] Step 5:

[0716] The converted output data is sent from the server to the 3D printing machine. The printing machine receives this data and outputs a model as a physical structure. Once the process from data transmission to model printing is complete, it is finally provided to the user.

[0717] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0718] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0719] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0720] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0721] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0722] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0723] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0724] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0725] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0726] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0727] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0728] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0729] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0730] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0731] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0732] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0733] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0734] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0735] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0736] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0737] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0738] The following is further disclosed regarding the embodiments described above.

[0739] (Claim 1)

[0740] A means of acquiring three-dimensional model data selected in a virtual environment,

[0741] A means for converting the three-dimensional model data into a specific output format using artificial intelligence that generates the data,

[0742] A means of transmitting the converted data to a three-dimensional printing device and outputting it as a physical object,

[0743] A system that includes this.

[0744] (Claim 2)

[0745] The system according to claim 1, further comprising means for optimizing the converted data and improving output efficiency.

[0746] (Claim 3)

[0747] The system according to claim 1, further comprising means for selecting output infrastructure and associated printing services.

[0748] "Example 1"

[0749] (Claim 1)

[0750] A means of acquiring data of an object model selected in a virtual environment via a terminal operated by the user,

[0751] A means for analyzing the data of the object model and converting it into a specific printable format using a machine learning algorithm,

[0752] A means of transmitting the converted data to a printing device and materializing it as an object,

[0753] A system that includes this.

[0754] (Claim 2)

[0755] The system according to claim 1, further comprising means for optimizing the converted data and improving the efficiency of the printing process.

[0756] (Claim 3)

[0757] The system according to claim 1, further comprising means for selecting equipment for output and associated printing functions.

[0758] "Application Example 1"

[0759] (Claim 1)

[0760] A means of acquiring three-dimensional representation data selected in a virtual space,

[0761] A means for converting the three-dimensional representation data into a specific output format using a machine learning model that generates the data,

[0762] A means for transmitting the converted data to a manufacturing device and outputting it as a physical item,

[0763] A means of acquiring and reflecting information on the selection and modification of items in a virtual space using a visual input device,

[0764] A system that includes this.

[0765] (Claim 2)

[0766] The system according to claim 1, further comprising means for optimizing the converted data and improving output efficiency.

[0767] (Claim 3)

[0768] The system according to claim 1, further comprising means for selecting manufacturing infrastructure and related manufacturing services.

[0769] "Example 2 of combining an emotion engine"

[0770] (Claim 1)

[0771] A means of analyzing sensor information to detect user emotions,

[0772] A means for presenting a selection of three-dimensional models based on detected emotional information,

[0773] A means of acquiring three-dimensional model data selected in a virtual environment,

[0774] A means for converting the three-dimensional model data into a specific output format using artificial intelligence that generates the data,

[0775] A means of transmitting the converted data to a three-dimensional printing device and outputting it as a physical object,

[0776] A system that includes this.

[0777] (Claim 2)

[0778] The system according to claim 1, further comprising means for optimizing the converted data and improving output efficiency.

[0779] (Claim 3)

[0780] The system according to claim 1, further comprising means for selecting output infrastructure and associated printing services.

[0781] "Application example 2 when combining with an emotional engine"

[0782] (Claim 1)

[0783] A means of analyzing the emotional state of users in a virtual space and recommending a three-dimensional model that is individually suited to them,

[0784] A means for converting the three-dimensional model data into a specific output format using artificial intelligence,

[0785] A means of transmitting the converted data to a 3D printing device and outputting it as a physical structure,

[0786] A means by which items are provided in an actual purchasing space in response to recommendations based on the emotions of interested individuals,

[0787] A system that includes this.

[0788] (Claim 2)

[0789] The system according to claim 1, further comprising means for optimizing the converted data and improving output efficiency.

[0790] (Claim 3)

[0791] The system according to claim 1, further comprising means for selecting an output base and associated printing services. [Explanation of symbols]

[0792] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of acquiring three-dimensional model data selected in a virtual environment, A means for converting the three-dimensional model data into a specific output format using artificial intelligence that generates the data, A means of transmitting the converted data to a three-dimensional printing device and outputting it as a physical object, A system that includes this.

2. The system according to claim 1, further comprising means for optimizing the converted data and improving output efficiency.

3. The system according to claim 1, further comprising means for selecting output infrastructure and associated printing services.

Images