System

Abstract

To provide a system for constructing a virtual environment and simulating the performance of an apparatus.SOLUTION: A system includes: means for receiving information on the performance of an apparatus which moves between a factory, a warehouse, and a shop; means generated on a virtual space showing the factory, the warehouse, and the shop; means for executing a simulation of movement of the apparatus by using a prompt sentence which instructs a simulation of movement of the apparatus on the virtual space and a generation AI model on the basis of performance data of the apparatus; and means for providing the user with the simulation result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Building a conventional 3D metaverse environment requires programming skills and specialized knowledge, and using that environment for business simulations requires even more complex settings and operations. These challenges limit the use of 3D metaverse environments and prevent them from fully realizing their potential. [Means for solving the problem]

[0005] This invention builds an elaborate 3D metaverse environment without requiring any programming or skills, creates 3D models from photographs, modifies them using text instructions, and inputs equipment performance (load capacity, speed, etc.). Furthermore, it generates entire factories, warehouses, stores, etc. in virtual space and simulates production lines, equipment operation, and people flow to derive optimal equipment performance, placement, and layout. This makes it easy to build and utilize a 3D metaverse environment, enabling it to be used more effectively in business simulations. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] In one embodiment of the present invention, a user first inputs a photograph into the system. This photograph may be, for example, a photograph of the interior or exterior of a factory, a product, or equipment. The system then automatically generates a 3D model from the photograph. The generated 3D model can be modified based on the user's text instructions. For example, the 3D model can be easily modified by inputting instructions into the system, such as "move equipment A two meters to the right" or "change the wall color to blue."

[0029] "Example 2"

[0030] Next, the user inputs the equipment's performance (load capacity, speed, etc.) into the system. This allows the system to accurately simulate the operation and performance of the equipment. For example, by inputting information such as the maximum load capacity of equipment A being 500 kg and the maximum speed being 1 m / s into the system, it is possible to perform a simulation that takes these performances into account. "Example 3"

[0031] Furthermore, users can generate entire factories, warehouses, stores, etc. in a virtual space, allowing them to visually grasp the overall layout and placement. Furthermore, by simulating the operation of production lines and equipment, as well as people flow, it is possible to derive optimal equipment performance, placement, and layout. For example, the system can automatically generate proposals for optimizing store layouts based on the results of people flow simulations.

[0032] The processing flow of each embodiment will be described below.

[0033] "Example 1"

[0034] Step 1: The user enters photos of the factory interior, exterior, products, equipment, etc. into the system.

[0035] Step 2: The system automatically generates a 3D model from the input photo.

[0036] Step 3: The user modifies the generated 3D model based on text instructions, such as "move device A 2 meters to the right" or "color the wall blue."

[0037] "Example 2"

[0038] Step 1: The user inputs the equipment's performance (load capacity, speed, etc.) into the system.

[0039] Step 2: The system simulates the operation and performance of the equipment based on the input performance information. For example, input information such as the maximum load capacity of Equipment A being 500 kg and the maximum speed being 1 m / s into the system.

[0040] "Example 3"

[0041] Step 1: The user generates an entire factory, warehouse, store, etc. in a virtual space.

[0042] Step 2: The system simulates the operation of production lines, equipment, and people flow in the generated virtual space.

[0043] Step 3: Based on the simulation results, the system derives optimal equipment performance, placement, layout, etc. For example, it automatically generates proposals for optimizing store layouts based on the results of a people flow simulation.

[0044] Example 1

[0045] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0046] Conventional 3D model generation systems require specialized programming and skills, making them difficult for general users to use. Furthermore, there were limitations on the accuracy of creating 3D models from photographs and the flexibility of modification based on user instructions. Furthermore, there was a lack of a way to feed back simulation results based on the generated 3D models into actual business operations.

[0047] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0048] In this invention, the server includes: means for inputting images captured by a user; means for the server to analyze the input images; means for the server to generate a three-dimensional model based on the analyzed data; means for the user to input text instructions; means for the server to analyze the text instructions and modify the three-dimensional model; means for inputting equipment performance; means for generating an entire factory, warehouse, store, etc. in a virtual space; means for simulating the operation of production lines and equipment and the flow of people; and means for deriving optimal equipment performance, placement, layout, etc. This allows users without specialized skills to easily generate and flexibly modify highly accurate three-dimensional models. Furthermore, the simulation results can be fed back into actual business operations.

[0049] A "user" is an individual or corporation that uses the system to input images and generate or modify three-dimensional models.

[0050] A "server" is a computer system that analyzes images input by a user and generates and modifies three-dimensional models.

[0051] "Images" refer to photographs or images taken by users and entered into the system.

[0052] "Analysis" means that the server extracts feature points, object contours, colors, etc. from the input image and processes them as data.

[0053] A "three-dimensional model" is a three-dimensional digital representation generated based on analyzed data.

[0054] A "text instruction" is a sentence that describes the details of the corrections or changes that the user will make to the three-dimensional model.

[0055] "Natural language processing" is a technology that allows a server to understand text instructions from a user and make appropriate corrections.

[0056] A "virtual space" is a simulated environment that digitally recreates the entirety of a factory, warehouse, store, etc.

[0057] "Simulation" is a technology that reproduces production lines, equipment operations, people flow, etc. in a virtual space to derive optimal placement and layout.

[0058] "Devices" are digital data such as three-dimensional models and simulation results generated by the system.

[0059] "Performance" refers to the physical characteristics of the equipment, such as load capacity and speed.

[0060] The present invention is a system that generates a three-dimensional model based on an image taken by a user and then modifies the model based on text instructions from the user. A specific embodiment of this system will be described below.

[0061] First, the user uses a terminal to upload images of the interior and exterior of the factory, products, equipment, etc. to the system. These images are in common image formats such as JPEG and PNG. The user opens the terminal's browser, accesses the system's upload page, clicks the "Select File" button to select the image to upload, and clicks the "Upload" button to send the image to the system.

[0062] Next, the server receives the uploaded image and analyzes it using image processing software (e.g., OpenCV or TENSORFLOW (registered trademark)). The server receives the image data and starts the image processing software to analyze the image's feature points, object contours, color, etc.

[0063] Based on the analyzed data, the server automatically generates a 3D model using 3D model generation software (e.g., Blender or Autodesk Maya). The server inputs the analysis results into the 3D model generation software, which then constructs a 3D model based on the image information. The generated 3D model is stored on the server and is ready to be displayed to the user.

[0064] The user checks the generated 3D model in the browser on their device and enters specific instructions for the parts they want to modify in the text boxes. For example, they might say, "Move device A two meters to the right" or "Change the wall color to blue." When the user clicks the "Submit" button, the instructions are sent to the system.

[0065] The server analyzes the text instructions from the user using natural language processing software (e.g., GPT-4 (registered trademark) or BERT) to understand the content of the instructions. The server receives the text instructions, launches the natural language processing software, analyzes the text instructions, and extracts specific corrections.

[0066] Based on the understood instructions, the server again uses the 3D model generation software to modify the 3D model. The server inputs the analysis results into the 3D model generation software, which modifies the 3D model based on the instructions. The modified 3D model is stored on the server and displayed to the user.

[0067] For example, if a user uploads an image of the inside of a factory to the system, the server receives the image, analyzes it using OpenCV, and generates a 3D model using Blender. Next, if the user inputs the instruction "move equipment A 2 meters to the right," the server analyzes this instruction using GPT-4 and again uses Blender to move equipment A in the 3D model 2 meters to the right.

[0068] An example of a prompt is as follows:

[0069] "Upload a photo of the interior of your factory. Then, if you want to make any modifications to the generated 3D model, enter specific instructions in text. For example, 'Move equipment A two meters to the right,' or 'Change the color of the walls to blue.'"

[0070] The above is a specific embodiment for carrying out the present invention.

[0071] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0072] Step 1:

[0073] A user uploads an image using a device.

[0074] Specific operation: The user opens the browser on the device, accesses the system's upload page, clicks the "Select file" button, selects the image to upload, and clicks the "Upload" button.

[0075] Input: User-taken image (JPEG or PNG format)

[0076] Output: Image data sent to the server

[0077] Step 2:

[0078] The server receives the uploaded images and analyzes them using image processing software.

[0079] How it works: The server receives the image data and launches image processing software such as OpenCV or TensorFlow, which analyzes the image's features, object contours, color, etc.

[0080] Input: Uploaded image data

[0081] Output: Analyzed image data (information such as feature points, contours, and colors)

[0082] Step 3:

[0083] The server generates a three-dimensional model based on the analyzed data using three-dimensional model generation software.

[0084] How it works: The server inputs the analysis results into 3D model generation software such as Blender or Autodesk Maya, which then creates a 3D model based on the image information.

[0085] Input: Analyzed image data

[0086] Output: Generated 3D model

[0087] Step 4:

[0088] The user uses a terminal to check the generated three-dimensional model and inputs correction instructions in text.

[0089] Specific operation: The user checks the 3D model in the browser on their device, enters specific instructions about the parts they want to correct in the text box, and clicks the "Submit" button.

[0090] Input: Generated 3D model, user text instructions

[0091] Output: Text instructions sent to the server

[0092] Step 5:

[0093] The server analyzes the text instructions from the user using natural language processing software.

[0094] Specific operation: The server receives the text instructions and launches natural language processing software such as GPT-4 or BERT. The software analyzes the text instructions and extracts specific corrections.

[0095] Input: User's text instructions

[0096] Output: Parsed modifications

[0097] Step 6:

[0098] The three-dimensional model generation software is then used again to modify the three-dimensional model based on the instructions understood by the server.

[0099] Specific operation: The server inputs the analysis results into the 3D model generation software. The software modifies the 3D model based on the instructions. The server saves the modified 3D model and displays it to the user.

[0100] Input: Analyzed corrections, generated 3D model

[0101] Output: Modified 3D model

[0102] The above is the specific flow of the program processing of this system.

[0103] (Application example 1)

[0104] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0105] Conventionally, optimizing the layout and equipment placement of factories, warehouses, stores, etc. required specialized knowledge and skills, and was problematic in that it was time-consuming and costly. Furthermore, there was no system that could generate a 3D model from on-site photos and modify it based on user instructions. This made it difficult to create a virtual environment that accurately reflected on-site conditions and perform efficient simulations.

[0106] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0107] In this invention, the server includes means for constructing an elaborate 3D virtual environment without requiring programming or skills, means for creating a 3D model from a photograph, means for making modifications based on text instructions, means for inputting equipment performance (load capacity, speed, etc.), means for generating entire factories, warehouses, stores, etc. in a virtual space, means for simulating the operation of production lines and equipment and the flow of people, means for deriving optimal equipment performance, placement, layout, etc., means for taking photographs of the inside of a factory using a smartphone and generating a 3D model from the photograph, means for modifying the generated 3D model based on the user's text instructions, and means for displaying the modified 3D model. This allows users to generate 3D models based on photos of the site and easily modify them, enabling efficient optimization of layout and equipment placement.

[0108] "A means of creating an elaborate 3D virtual environment without the need for programming or skills" is a method that allows even users without specialized knowledge or skills to easily create a highly accurate 3D virtual space.

[0109] "Means for creating 3D models from photographs" refers to a technology that automatically generates 3D models based on photographs taken by users.

[0110] The "means for modifying by text instructions" is a method for modifying a 3D model based on text instructions entered by a user.

[0111] "Means for inputting equipment performance (load capacity, speed, etc.)" refers to a method for inputting performance and characteristics into the generated 3D model.

[0112] "Means for generating entire factories, warehouses, stores, etc. in virtual space" refers to technology that reproduces actual spaces such as factories, warehouses, and stores in virtual space.

[0113] "Means for simulating the operation of production lines and equipment, and the flow of people" refers to a method for simulating the operation of production lines and equipment, and the flow of people in a virtual space.

[0114] "Means for deriving optimal equipment performance, placement, layout, etc." refers to a method for determining optimal equipment performance, placement, and layout based on simulation results.

[0115] "A means of taking photos of the inside of a factory using a smartphone and generating a 3D model from those photos" is a technology that generates a 3D model based on photos of the inside of a factory taken with a smartphone.

[0116] The "means for modifying the generated 3D model based on the user's text instructions" is a method for modifying the generated 3D model according to the text instructions entered by the user.

[0117] The "means for displaying the modified 3D model" is a technology for visually displaying the modified 3D model.

[0118] A system for implementing this invention is configured as follows: First, a user takes a photo of the inside of a factory using a smartphone. The taken photo is input into the system, which automatically generates a 3D model from the photo. The generated 3D model can be modified based on the user's text instructions. For example, possible instructions include "move machine A 3 meters to the left" or "change the color of the wall to green." The modified 3D model is visually displayed, allowing the user to confirm the results.

[0119] This system is implemented using the following hardware and software: a smartphone, a server, and a display device. The software uses OpenCV, Blender, and OpenAI's GPT-3.

[0120] The specific process is as follows: First, a photo taken with a smartphone is sent to the server. The server analyzes the photo using OpenCV and generates a 3D model. The generated 3D model is visually displayed using Blender. When the user enters text instructions, the instructions are sent to the server and analyzed using OpenAI's GPT-3. Based on the analysis results, the 3D model is modified and displayed again in Blender.

[0121] As a concrete example, consider a scenario in which the layout of machines in a factory needs to be changed. A user takes a photo of the factory and inputs the instruction "move machine A 3 meters to the left." The system generates a 3D model from the photo and repositions machine A based on the instruction. The modified 3D model is visually displayed to the user, allowing the user to confirm the results.

[0122] An example of a prompt is as follows:

[0123] "Take a photo of your factory and enter it into the app. Then enter instructions like this:

[0124] "Move machine A 3 meters to the left"

[0125] Change the wall color to green

[0126] "Move product B forward 2 meters."

[0127] In this way, users can easily optimize the layout of their factory.

[0128] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0129] Step 1:

[0130] A user takes a photo of the inside of a factory using a smartphone. The photo is saved in the smartphone. The input is the photo of the inside of the factory, and the output is the image file saved in the smartphone.

[0131] Step 2:

[0132] A user starts the application on their smartphone and uploads a photograph they have taken to the system. The input is the image file on the smartphone, and the output is the image data sent to the server.

[0133] Step 3:

[0134] The server analyzes the received image data using OpenCV and generates a 3D model from the photo. The input is the image data sent to the server, and the output is the generated 3D model data. Specifically, it extracts feature points from the image and reconstructs the 3D shape based on them.

[0135] Step 4:

[0136] The server visually displays the generated 3D model using Blender. The input is the generated 3D model data, and the output is the visually displayed 3D model. Specifically, it uses Blender's API to render the 3D model.

[0137] Step 5:

[0138] The user inputs text instructions through the application, for example, "move machine A 3 meters to the left." The input is the user's text instructions, and the output is the text data sent to the server.

[0139] Step 6:

[0140] The server uses OpenAI's GPT-3 to analyze the received text instructions. The input is the text data sent to the server, and the output is the analysis result. Specifically, the text instructions are processed using the GPT-3 API for natural language processing.

[0141] Step 7:

[0142] The server modifies the 3D model based on the analysis results. The input is the analysis results and the generated 3D model data, and the output is the modified 3D model data. Specifically, it uses Blender's API to change the position and color of the 3D model.

[0143] Step 8:

[0144] The server visually displays the modified 3D model again using Blender. The input is the modified 3D model data, and the output is the visually displayed modified 3D model. Specifically, the server renders the modified 3D model using Blender's API.

[0145] Step 9:

[0146] The user checks the corrected 3D model through the application. The input is the visually displayed corrected 3D model, and the output is the user's confirmation result. Specifically, the corrected 3D model is displayed on the smartphone display.

[0147] Example 2

[0148] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0149] In conventional systems, advanced programming skills were required to build an elaborate virtual environment, and complex settings were often required to run simulations based on equipment performance data. This made it difficult for users to easily build a virtual environment and simulate equipment performance. Furthermore, there was a lack of a way to feed back the simulation results into actual business operations.

[0150] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0151] In this invention, the server includes means for receiving information on the performance of equipment moving between factories, warehouses, and stores, means for generating a virtual space representing the factories, warehouses, and stores, a prompt statement for instructing a simulation of the movement of the equipment in the virtual space based on the performance data of the equipment, means for executing a simulation of the movement of the equipment using a generative AI model, and means for providing a user with the simulation results. This enables a user without advanced skills to build a virtual environment, simulate the performance of equipment, and reflect the results in actual business operations. The generating means in the server generates a three-dimensional model representing the factory, warehouse, and store in a virtual space from photographs representing the factory, warehouse, and store. The executing means in the server executes a simulation of the movement of the equipment using the prompt sentence, the generating AI model, and the simulation software.

[0152] A "virtual environment" is a digital space generated on a computer that mimics the real world.

[0153] A "three-dimensional model" is digital data that represents the shape and structure of an object in three-dimensional space.

[0154] A "text instruction" is an instruction for an operation or correction given to the system by the user using text information.

[0155] "Equipment performance data" refers to data that indicates the physical characteristics, such as the load capacity and speed, of equipment moving between factories, warehouses, and stores.

[0156] A "generative AI model" is a model that uses artificial intelligence technology to set up simulations based on equipment performance data.

[0157] A "prompt" is a text sentence that instructs the generative AI model on the simulation settings and conditions.

[0158] "Simulation" is the process of simulating the operation and performance of equipment in a virtual environment.

[0159] "Feedback" is the process of reflecting the results of the simulation in actual business operations.

[0160] This invention is a system in which a user inputs performance data of a device, a server uses a generated AI model to perform a simulation based on that data, and provides the results to the user. A specific embodiment of this system will be described below.

[0161] Hardware and software used

[0162] Hardware: Servers, user terminals

[0163] Software: Generative AI models, simulation software

[0164] System processing flow

[0165] First, three-dimensional models representing the factories, warehouses, and stores are generated in a virtual space from photographs of the factories, warehouses, and stores in the same manner as described in the first embodiment. Then, the following processing is performed. 1. The user inputs the device performance data.

[0166] The user inputs the performance data of the device into the input form on the terminal. For example, the maximum load capacity of device A is 500 kg, and the maximum speed is 1 m / s.

[0167] 2. The device sends the entered performance data to the server.

[0168] The terminal converts the performance data entered by the user into a data format such as JSON and sends an HTTP request to the server.

[0169] 3. The server receives the performance data and generates a prompt for the generative AI model.

[0170] The server analyzes the received performance data and generates a prompt to input to the generative AI model. For example, it generates a prompt such as, "Please perform an operation simulation when the maximum load capacity of device A is 500 kg and the maximum speed is 1 m / s."

[0171] 4. The server uses the generated AI model to set up the simulation.

[0172] The server inputs a prompt statement into the generative AI model to generate the settings necessary for the simulation. The generative AI model then sets the simulation parameters and conditions based on the prompt statement.

[0173] 5. The server runs the simulation using the simulation software.

[0174] The server launches the simulation software and runs a simulation based on the settings obtained from the generative AI model. The simulation software calculates the behavior and performance of the equipment and generates simulation results.

[0175] 6. The server sends the simulation results to the device.

[0176] The server converts the simulation results into a data format such as JSON and sends them to the terminal as an HTTP response.

[0177] 7. The device displays the simulation results to the user.

[0178] The device analyzes the received simulation results and displays them in a format that is easy for the user to view. For example, the simulation results may show, in graphs and text, how device A behaves when it supports a load of 500 kg or when it operates at a speed of 1 m / s.

[0179] Specific examples

[0180] Prompt Sentence Examples

[0181] "Please perform a simulation of the operation of device A when the maximum load capacity is 500 kg and the maximum speed is 1 m / s."

[0182] Simulation result display example

[0183] "Device A can support a load of 500 kg and has been confirmed to operate stably at a speed of 1 m / s."

[0184] In this way, the server uses the generated AI model and simulation software to simulate the operation and performance of equipment based on performance data entered by the user, and provides the results to the user.This system enables users to create a virtual environment, simulate equipment performance, and reflect the results in actual business operations without requiring advanced skills.

[0185] The flow of the identification process in the second embodiment will be described with reference to FIG. First, a three-dimensional model representing the factory, warehouse, and store is generated in a virtual space from photographs of the factory, warehouse, and store. Then, the following steps are performed.

[0186] Step 1:

[0187] The user inputs the performance data of the device.

[0188] Input: The user enters the equipment's performance data (e.g., maximum load capacity of 500 kg, maximum speed of 1 m / s) into the terminal's input form.

[0189] Specific operation: The user uses the keyboard to enter device performance data into a form displayed on the terminal screen.

[0190] Step 2:

[0191] The terminal transmits the input performance data to the server.

[0192] Input: User-entered performance data.

[0193] Output: JSON formatted data sent to the server.

[0194] Specific operation: The terminal converts the performance data entered by the user into JSON format, creates an HTTP request, and sends it to the server.

[0195] Step 3:

[0196] The server receives the performance data and generates prompt sentences for the generative AI model.

[0197] Input: Performance data sent from the device in JSON format.

[0198] Output: A prompt to be fed into the generative AI model.

[0199] Specific operation: The server analyzes the received performance data and generates a prompt message saying, "Please perform an operation simulation when the maximum load capacity of device A is 500 kg and the maximum speed is 1 m / s."

[0200] Step 4:

[0201] The server uses the generated AI model to set up the simulation.

[0202] Input: The generated prompt statement.

[0203] Output: The configuration parameters required for the simulation.

[0204] Specific operation: The server inputs a prompt statement into the generative AI model, and the generative AI model sets the simulation parameters and conditions based on the prompt statement.

[0205] Step 5:

[0206] The server runs the simulation using the simulation software.

[0207] Input: Simulation setup parameters obtained from the generative AI model.

[0208] Output: Simulation results.

[0209] Specific operation: The server starts the simulation software, inputs the setting parameters, and runs the simulation. The simulation software calculates the behavior and performance of the equipment and generates the simulation results.

[0210] Step 6:

[0211] The server transmits the simulation results to the terminal.

[0212] Input: Simulation results.

[0213] Output: Simulation result data in JSON format sent to the terminal.

[0214] Specific operation: The server converts the simulation results into JSON format and sends them to the terminal as an HTTP response.

[0215] Step 7:

[0216] The terminal displays the simulation results to the user.

[0217] Input: Simulation result data in JSON format sent from the server.

[0218] Output: Simulation results that are displayed to the user.

[0219] Specific operation: The device analyzes the received simulation results and displays them in a format that is easy for the user to view. For example, the simulation results may show the behavior of device A when it supports a load of 500 kg or when it operates at a speed of 1 m / s, as graphs or text.

[0220] (Application example 2)

[0221] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0222] With conventional systems, building virtual environments for factories, warehouses, stores, etc. required programming and specialized skills, making them difficult for general users to use. Furthermore, there was a lack of means to accurately simulate equipment performance, making it difficult to derive optimal equipment performance and placement for actual operation. Furthermore, there was no way to visually check the simulation results, making it difficult for users to intuitively understand the results.

[0223] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0224] In this invention, the server includes means for constructing an elaborate 3D virtual environment without requiring programming or skills, means for creating a 3D model from an image, means for making modifications using text instructions, means for inputting equipment performance (load capacity, speed, etc.), means for generating entire factories, warehouses, stores, etc. in a virtual space, means for simulating the operation and flow of production lines and equipment, means for deriving optimal equipment performance, placement, layout, etc., and means for simulating and visually displaying the operation and performance of robots based on performance data input by the user. This enables users to easily construct a virtual environment without specialized skills, accurately simulate equipment performance, and visually confirm it.

[0225] A "3D virtual environment" is a three-dimensional virtual space generated on a computer, which users can experience visually.

[0226] "Means for creating 3D models from images" refers to a technology that generates three-dimensional solid models based on two-dimensional image data.

[0227] "Means for making corrections based on text instructions" is a technology that automatically makes corrections to the virtual environment or model based on text information entered by the user.

[0228] The "means for inputting equipment performance" is an interface that allows the user to input equipment characteristics (such as load capacity and speed) into the system.

[0229] "Means of generating in virtual space" refers to technology that reproduces real-world environments such as factories, warehouses, and stores in virtual space.

[0230] "Means of simulation" refers to technology that calculates behavior and performance within a virtual environment and mimics actual behavior.

[0231] "Means for deriving optimal equipment performance, placement, layout, etc." refers to technology that calculates the most efficient equipment performance, placement, and layout based on simulation results.

[0232] "Visual display means" refers to technology that graphically displays simulation results, allowing users to intuitively understand them.

[0233] This invention is a system that allows users without specialized skills to easily create a 3D virtual environment, accurately simulate the performance of equipment, and visually check it. A specific embodiment of this system is described below.

[0234] System configuration

[0235] The system consists of the following main components:

[0236] 1. Server: Manages the entire system and processes data.

[0237] 2. User terminal: Provides an interface for user operation.

[0238] 3. Software: Programs for building, simulating and visualizing 3D virtual environments.

[0239] Hardware and software used

[0240] Hardware: Regular PC or tablet

[0241] Software: Python, matplotlib, numpy

[0242] Program processing explanation

[0243] The server receives performance data (such as load capacity and speed) entered by the user and performs a simulation based on this data. Specifically, the process proceeds as follows:

[0244] 1. User input: The user inputs the device's performance data through a terminal, such as "maximum load capacity 500 kg, maximum speed 1 m / s."

[0245] 2. Data processing: The server performs a simulation based on the received performance data. It uses the Python numpy library to calculate the change in speed and load over time.

[0246] 3. Visualization: The simulation results are displayed graphically using the matplotlib library, which allows users to intuitively understand the results.

[0247] Specific examples

[0248] For example, if a user wants to simulate a robot with a maximum load capacity of 500 kg and a maximum speed of 1 m / s, they would enter the following:

[0249] python

[0250] max_load = 500 kg

[0251] max_speed = 1 m / s

[0252] The server runs a simulation based on this data and visually displays the results, allowing users to intuitively understand the robot's behavior and performance.

[0253] Prompt Sentence Examples

[0254] "Please simulate the movement of a robot with a maximum load capacity of 500 kg and a maximum speed of 1 m / s."

[0255] In this way, users can easily create a virtual environment, simulate the performance of equipment, and visually check it, even if they do not have specialized skills.

[0256] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0257] Step 1:

[0258] The user uses a terminal to input the performance data of the equipment (for example, maximum load capacity and maximum speed). The input data is sent to the server. An example of input data is "maximum load capacity 500 kg, maximum speed 1 m / s."

[0259] Step 2:

[0260] The server prepares the simulation based on the received performance data. Specifically, it uses the Python numpy library to generate the time axis data required for the simulation. Based on the input data, time axis data (for example, 100 steps from 0 seconds to 10 seconds) is generated.

[0261] Step 3:

[0262] The server calculates the change in velocity and load based on the generated time axis data. Specifically, it uses the numpy library to calculate the change in velocity and load over time. For example, the velocity is calculated to increase linearly with time, and the load is calculated to increase exponentially with time.

[0263] Step 4:

[0264] The server generates simulation results based on the calculation results. Specifically, it uses the calculated velocity and load data and stores the simulation results in a data structure. The simulation results include velocity and load values ​​at each time step.

[0265] Step 5:

[0266] The server generates graphs using the matplotlib library to visualize the simulation results. Specifically, it plots the changes in speed and load against time and creates a graph to visually display them. The generated graph visually shows the changes in speed and load.

[0267] Step 6:

[0268] The server sends the generated graphs to the user's device, where the user can visually check the simulation results, allowing them to intuitively understand the robot's behavior and performance.

[0269] Step 7:

[0270] Based on the visualized simulation results, users can re-enter performance data as needed and run the simulation again, enabling iterative simulations to derive optimal equipment performance and placement.

[0271] Example 3

[0272] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0273] Determining efficient layouts and equipment placements is extremely important in modern physical spaces such as factories, warehouses, and stores. However, achieving this requires advanced specialized knowledge and skills, and repeated trial and error can be time-consuming and costly. Furthermore, there is a lack of means to analyze simulation results and derive optimal placements and layouts, making efficient operation difficult.

[0274] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0275] In this invention, the server includes means for constructing an elaborate 3D virtual environment without requiring programming or skills, means for creating a 3D model from an image, means for making modifications via text instructions, means for inputting equipment performance (load capacity, speed, etc.), means for generating entire factories, warehouses, stores, etc. in a virtual space, means for simulating the operation of production lines and equipment and the flow of people, means for deriving optimal equipment performance, placement, layout, etc., means for using a generative AI model to analyze the simulation results, and means for automatically generating optimization proposals based on the analysis results. This enables users to determine efficient layouts and equipment placements without requiring advanced specialized knowledge or skills.

[0276] A "3D virtual environment" is a three-dimensional virtual space generated on a computer, in which users can visually recreate and manipulate physical space.

[0277] "Means for creating 3D models from images" refers to a technology that generates three-dimensional models based on two-dimensional image data, and is a means that can create three-dimensional objects from photographs or drawings.

[0278] "Means for modifying by text instructions" refers to a technique that allows a user to modify objects and layouts in a virtual environment by inputting instructions in text format.

[0279] "Means for inputting equipment performance" refers to a technique for inputting performance data (for example, load capacity, speed, etc.) of objects and equipment used in a virtual environment.

[0280] "A means of generating entire factories, warehouses, stores, etc. in virtual space" is a technology that reproduces the entire physical space in a three-dimensional virtual environment, allowing users to visually grasp that space.

[0281] "Means for simulating the operation of production lines and equipment, and the flow of people" refers to technology that simulates the operation of production lines, the movement of equipment, and the flow of people within a virtual environment, and is a means that can simulate actual operations.

[0282] "Means for deriving optimal equipment performance, placement, layout, etc." refers to technology that automatically proposes optimal equipment performance, placement, and layout based on simulation results.

[0283] "Means using generative AI models" refers to technology that uses artificial intelligence models to analyze data and generate results, such as means for analyzing simulation results and proposing optimization.

[0284] "Means for automatically generating optimization proposals based on analysis results" refers to technology that automatically generates proposals for optimal placement and layout based on the analysis results obtained by a generative AI model.

[0285] This invention provides a system that allows a user to generate a physical space such as a factory, warehouse, or store in a virtual space and determine an efficient layout and arrangement of equipment. Specific embodiments of this system are described below.

[0286] Hardware and software used

[0287] Hardware: PC with a high-performance graphics card, VR headset

[0288] Software: Unity (game engine), Python (programming language), generative AI models (e.g., OpenAI's GPT-4)

[0289] Virtual space generation

[0290] Users use Unity to recreate physical spaces such as factories, warehouses, and stores in a virtual space. Specifically, they use Unity's editor to design the building structure and internal layout. For example, they can set the placement of walls, floors, and equipment using drag and drop.

[0291] Entering data

[0292] Users use Python scripts to input production line and equipment operation and people flow data. Specifically, people flow data stored in a CSV file is read using Python and passed to Unity. For example, the pandas library is used to read the CSV file and format the data.

[0293] Running the simulation

[0294] The server runs the simulation on Unity. Specifically, it launches Unity's simulation engine and visualizes the operation of the production line and the movement of people in real time based on the data entered by the user. For example, it displays the operation of equipment and the movement of people in 3D models.

[0295] Analysis of simulation results

[0296] The server uses a generative AI model (e.g., GPT-4) to analyze the simulation results. Specifically, the simulation results are saved in a log file and the data is input into the generative AI model. For example, the people flow data during the simulation is saved in text format and analyzed by GPT-4.

[0297] Optimization suggestions

[0298] Based on the analysis results, the server automatically generates proposals for optimal equipment performance, placement, and layout. Specifically, it presents the optimal layout and placement to the user based on the analysis results obtained from the generative AI model. For example, it makes specific suggestions such as, "Since foot traffic is concentrated in this area, we suggest changing the placement of the cash registers."

[0299] Specific examples

[0300] Factory example

[0301] Virtual space generation: Users design the layout of a factory in Unity, for example, by setting up the layout of production lines and equipment.

[0302] Data input: Users input production line operation data in a Python script, for example, by reading operation data from a CSV file.

[0303] Simulation execution: The server runs the simulation in Unity to visualize the operation of the production line, for example by displaying a 3D model of the equipment in operation.

[0304] Analysis of simulation results: The server uses the generative AI model to analyze the simulation results. For example, the server saves the motion data in text format and runs GPT-4 to analyze it.

[0305] Optimization proposal: The server proposes the optimal layout of the production line, for example, making specific proposals such as "By changing the layout of this equipment, efficiency will improve by 10%."

[0306] Store example

[0307] Creating a virtual space: Users design the layout of a store in Unity, for example, by setting the placement of shelves and cash registers.

[0308] Data input: Users input people flow data in a Python script, for example, by reading people flow data from a CSV file.

[0309] Simulation execution: The server runs the simulation in Unity and visualizes the movement of people, for example, displaying a 3D model of how customers move through a store.

[0310] Analysis of simulation results: The server uses the generative AI model to analyze the simulation results. For example, the server saves the people flow data in text format and runs GPT-4 to analyze it.

[0311] Optimization suggestions: The server will suggest optimal product placement, such as "Since there is a high concentration of people in this area, we suggest changing the placement of popular products."

[0312] Prompt Sentence Examples

[0313] Factory Optimization

[0314] To optimize your factory layout, input production line operation data. Based on the simulation results, we will suggest the optimal layout.

[0315] Store optimization

[0316] Enter foot traffic data to optimize your store layout. Based on the simulation results, we will suggest optimal product placement.

[0317] The flow of the identification process in the third embodiment will be described with reference to FIG.

[0318] Step 1:

[0319] Users use Unity to recreate physical spaces such as factories, warehouses, and stores in virtual spaces.

[0320] Input: User-designed building structure and interior layout data.

[0321] How it works: The user opens the Unity editor and drags and drops the placement of walls, floors, and equipment.

[0322] Output: A 3D model reproduced in virtual space.

[0323] Step 2:

[0324] Users use Python scripts to input production line and equipment operation and people flow data.

[0325] Input: Production line and equipment operation data, and people flow data in a CSV file.

[0326] Specific behavior: The user uses the pandas library to read the CSV file, format the data, and pass it to Unity.

[0327] Output: The formatted data passed to Unity.

[0328] Step 3:

[0329] The server runs the simulation on Unity.

[0330] Input: Production line and equipment operation data, and people flow data entered by users.

[0331] How it works: The server runs Unity's simulation engine and visualizes the operation of the production line and the movement of people in real time, for example, displaying 3D models of how equipment operates and how people move.

[0332] Output: Log file of simulation results.

[0333] Step 4:

[0334] The server uses the generative AI model to analyze the simulation results.

[0335] Input: Simulation results log file.

[0336] Specific operation: The server saves the simulation results in text format and inputs the data into a generative AI model. For example, GPT-4 analyzes the people flow data during the simulation.

[0337] Output: Analysis result data.

[0338] Step 5:

[0339] Based on the analysis results, the server automatically generates proposals for optimal equipment performance, placement, and layout.

[0340] Input: Data from the analysis results obtained from the generative AI model.

[0341] Specific operation: Based on the analysis results, the server presents the user with the optimal layout and placement. For example, it makes a specific suggestion such as, "Since there is a high concentration of people in this area, we suggest changing the placement of the cash registers."

[0342] Output: Optimization suggestions.

[0343] (Application example 3)

[0344] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0345] In conventional store operations, it was difficult to optimize layout and product placement, hindering efficient operations. Furthermore, there was a lack of means to grasp the actual flow of people in the store in real time and instantly change the layout. This resulted in poor customer flow, which could lead to lower sales and lower customer satisfaction.

[0346] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes means for constructing an elaborate 3D virtual environment without requiring programming or skills, means for creating a 3D model from a photograph, means for making modifications using text instructions, means for inputting equipment performance (load capacity, speed, etc.), means for generating an entire factory, warehouse, store, etc. in a virtual space, means for simulating the operation of production lines and equipment and people flow, means for deriving optimal equipment performance, placement, layout, etc., means for proposing optimal product placement and layout based on the results of the people flow simulation in the virtual space, and means for checking and changing the store layout in real time using a smartphone, smart glasses, or head-mounted display. This allows for efficient optimization of store layout and product placement, improving customer flow, and increasing sales and customer satisfaction.

[0347] A "3D virtual environment" is a three-dimensional virtual space generated on a computer, which the user can visually grasp.

[0348] "Means for creating 3D models from photographs" refers to a technology that automatically generates three-dimensional models based on photographic data.

[0349] "Means for modifying by text instructions" refers to a technique that allows a user to modify objects and layouts in a virtual environment by inputting instructions in text format.

[0350] "Means for inputting equipment performance" refers to technology for inputting the performance (e.g., load capacity, speed, etc.) of the generated model or object.

[0351] "Means of generating in virtual space" refers to technology that reproduces actual facilities such as factories, warehouses, and stores in virtual space.

[0352] "Means for simulating the operation of production lines and equipment, and the flow of people" refers to technology that simulates the operation of production lines and equipment, and the flow of people, in a virtual space.

[0353] "Means for deriving optimal equipment performance, placement, layout, etc." refers to technology that calculates and proposes optimal equipment performance, placement, and layout based on simulation results.

[0354] "Means for proposing optimal product placement and layout based on the results of people flow simulation" refers to technology that automatically proposes optimal product placement and layout based on the results of people flow simulation in a virtual space.

[0355] "Means for checking and making changes to store layouts in real time using smartphones, smart glasses, and head-mounted displays" refers to technology that uses these devices to check store layouts in a virtual space in real time and make changes as necessary.

[0356] A system for implementing the present invention uses the following hardware and software.

[0357] Hardware

[0358] Smartphones (e.g., iPhone (registered trademark), ANDROID (registered trademark) devices)

[0359] Smart glasses (e.g., Google® Glass®, Microsoft® HoloLens®)

[0360] Head-mounted displays (e.g., Oculus Rift, HTC Vive)

[0361] software

[0362] Python 3.x

[0363] NumPy (numerical calculation library)

[0364] Matplotlib (data visualization library)

[0365] SciPy (optimized library)

[0366] System Overview

[0367] The server first provides a means for constructing elaborate 3D virtual environments without requiring programming or skills. This allows users to easily generate virtual spaces. Next, a means for creating 3D models from photographs is used to generate 3D models of actual stores and products. Finally, a means for making corrections using text instructions allows users to easily modify objects and layouts in the virtual space.

[0368] By using a means to input equipment performance (load capacity, speed, etc.), performance data for the generated models and objects can be input, which allows for the creation of entire factories, warehouses, stores, etc. in virtual space.

[0369] The server performs simulations in a virtual space using means for simulating production lines, equipment operations, and people flow. Based on the results of this simulation, a means is provided for deriving optimal equipment performance, placement, layout, etc. Furthermore, it includes means for proposing optimal product placement and layout based on the results of people flow simulation in a virtual space.

[0370] Using a smartphone, smart glasses, or head-mounted display, users can check the store layout in real time and make changes as needed, enabling efficient optimization of store layout and product placement, improving customer flow.

[0371] Specific examples

[0372] For example, when optimizing the layout of a supermarket, the system inputs the product layout as the initial layout and performs a people flow simulation. Based on the results, the system proposes the optimal product placement. Users can use their smartphones or smart glasses to check the layout in real time in the virtual space and make changes as necessary.

[0373] Prompt Sentence Examples

[0374] "Enter the initial layout of your supermarket, run a people flow simulation and suggest the optimal product placement."

[0375] In this way, the virtual store layout optimization assistant optimizes the layout of physical stores and supports efficient store operations.

[0376] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0377] Step 1:

[0378] The server receives initial layout data from the user. The user inputs product placement and store layout using a smartphone or PC. This input data becomes the base data for generating the virtual space.

[0379] Step 2:

[0380] The server generates a three-dimensional model based on the user-provided photo data using a means for creating a 3D model from a photo. This process uses image analysis technology to extract three-dimensional shapes from the photo and build a 3D model. The output is a 3D model that can be used in the virtual space.

[0381] Step 3:

[0382] The server uses a means for making corrections based on text instructions to correct objects and layouts in the virtual space based on text instructions entered by the user. The user inputs correction instructions in text format using a smartphone or PC. The server analyzes the instructions and corrects the corresponding parts in the virtual space. The output is the layout of the corrected virtual space.

[0383] Step 4:

[0384] The server inputs performance data provided by the user using a means for inputting equipment performance (load capacity, speed, etc.). This data is used for simulation in the virtual space. The input data includes the physical properties of each object. The output is a model of the virtual space that reflects the performance data.

[0385] Step 5:

[0386] The server uses a means to generate an entire virtual space, such as a factory, warehouse, or store, based on the data provided by the user. This process integrates the initial layout data, 3D models, and performance data to generate the virtual space. The output is a model of the complete virtual space.

[0387] Step 6:

[0388] The server performs simulations in a virtual space using means to simulate the operation of production lines, equipment, and people. In this process, the server simulates the operation of people and equipment in the virtual space based on data provided by the user. The output is the simulation results.

[0389] Step 7:

[0390] The server calculates the optimal placement using a method to derive the optimal equipment performance, placement, and layout based on the simulation results. In this process, the server analyzes the simulation results and proposes the optimal placement. The output is a proposal for an optimized layout.

[0391] Step 8:

[0392] The server uses a method to propose optimal product placement and layout based on the results of the virtual human flow simulation, and makes specific placement proposals. This process analyzes human flow data and calculates optimal product placement. The output is a proposal for optimal product placement.

[0393] Step 9:

[0394] Users use smartphones, smart glasses, or head-mounted displays to view the store layout in real time and make changes as needed. Using these devices, users visually view the layout in the virtual space and direct changes. The output is a layout of the virtual space updated in real time.

[0395] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0396] "Example 1"

[0397] In one embodiment of the present invention, a system incorporating an emotion engine is provided. The system recognizes a user's emotions and adjusts the 3D metaverse environment based on the user's emotions. Specifically, when the user feels joy, the system adjusts the 3D metaverse environment to a bright and cheerful atmosphere. Conversely, when the user feels sad, the system adjusts the 3D metaverse environment to a quiet and calm atmosphere.

[0398] "Example 2"

[0399] The emotion engine also adjusts the simulation results based on the user's emotions. For example, if a user feels stressed, the system will provide simulation results to alleviate the user's emotions, such as slowing down the production line. This allows users to experience the optimal environment according to their emotions.

[0400] "Example 3"

[0401] Furthermore, the emotion engine recognizes the user's emotions and adjusts the 3D metaverse environment based on those emotions. For example, when the user feels happy, the system changes the music in the 3D metaverse environment to a bright and cheerful one. When the user feels sad, the system changes the music in the 3D metaverse environment to a quiet and calm one. This allows the user to experience the optimal environment according to their emotions.

[0402] The processing flow of each embodiment will be described below.

[0403] "Example 1"

[0404] Step 1: The system recognizes the user's emotions using an emotion engine.

[0405] Step 2: Adjust the 3D metaverse environment based on the recognized emotion. Specifically, when the user feels joy, the system adjusts the 3D metaverse environment to a bright and happy atmosphere.

[0406] Step 3: Conversely, when the user feels sad, the system adjusts the 3D metaverse environment to a quiet and calming atmosphere.

[0407] "Example 2"

[0408] Step 1: The system recognizes the user's emotions using an emotion engine.

[0409] Step 2: Adjust the simulation results based on the recognized emotions. Specifically, when the user feels stressed, the system will provide simulation results to alleviate the user's emotions, such as slowing down the production line.

[0410] "Example 3"

[0411] Step 1: The system recognizes the user's emotions using an emotion engine.

[0412] Step 2: Adjust the 3D metaverse environment based on the recognized emotion. Specifically, when the user feels joy, the system changes the music in the 3D metaverse environment to a brighter, more cheerful one.

[0413] Step 3: Also, when the user feels sad, the system changes the music in the 3D Metaverse environment to a quieter, more soothing one.

[0414] Example 1

[0415] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0416] Previous systems for building 3D metaverse environments required programming and advanced skills, making them difficult for average users to use. Other issues included the accuracy of creating 3D models from photographs and the time required to modify the generated 3D models. Furthermore, the system lacked the functionality to adjust the 3D metaverse environment according to the user's emotions, creating a need for an improved user experience.

[0417] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for creating a 3D model from a photograph, a means for making modifications based on text instructions, and a means for recognizing a user's emotions and adjusting the 3D metaverse environment based on the emotions. This allows a user to generate an elaborate 3D model from a photograph and easily modify it based on text instructions without requiring programming or advanced skills. Furthermore, automatically adjusting the 3D metaverse environment according to the user's emotions can provide a more intuitive and comfortable user experience.

[0418] "Means for creating 3D models from photographs" refers to technology that analyzes photographs entered by users and automatically generates 3D models based on those photographs.

[0419] The "means for performing corrections based on text instructions" is a technology for correcting a generated 3D model based on text instructions entered by a user.

[0420] "Means for recognizing the user's emotions and adjusting the 3D Metaverse environment based on those emotions" refers to a technology that analyzes the user's emotions from facial expressions, voice, etc., and automatically adjusts the 3D Metaverse environment based on the results of that analysis.

[0421] "Means for inputting equipment performance" refers to a technology for inputting the performance of the generated 3D model (e.g., load-bearing capacity, speed, etc.) into the system.

[0422] "Means for generating entire factories, warehouses, stores, etc. in virtual space" refers to technology that reproduces physical spaces such as factories, warehouses, and stores in virtual space.

[0423] "Means for simulating the operation of production lines and equipment, and the flow of people" refers to technology that simulates the operation of production lines and equipment, and the flow of people in a virtual space.

[0424] "Means for deriving optimal equipment performance, placement, layout, etc." refers to technology that determines optimal equipment performance, placement, and layout based on simulation results.

[0425] This invention is a system that allows a user to input a photo, generate a 3D model from the photo, and further adjust the 3D Metaverse environment based on text instructions and emotions. Specific embodiments of this system are described below.

[0426] First, the user inputs photos of the interior and exterior of the factory, products, equipment, etc. into the terminal. The terminal then sends these photos to the server. The server analyzes the received photos and automatically generates a 3D model. This analysis and generation uses image processing software (e.g., "OpenCV") and 3D modeling software (e.g., "Blender").

[0427] Next, the user inputs text instructions for the generated 3D model, such as "move device A two meters to the right" or "change the wall color to blue." The device sends the text instructions to the server, which uses a generative AI model (e.g., "OpenAI GPT-4") to analyze the text instructions and modify the 3D model.

[0428] Furthermore, the system also includes a function to recognize the user's emotions and adjust the 3D metaverse environment based on those emotions. The user expresses their emotions to the system (e.g., facial expressions or voice). The device captures the user's facial expressions and voice and sends them to the server. The server analyzes the user's emotions using an emotion recognition engine (e.g., "Microsoft Azure (registered trademark) Emotion API"). Based on the analysis results, the server adjusts the 3D metaverse environment. For example, if the user is feeling happy, the environment will be adjusted to have a bright and cheerful atmosphere.

[0429] As a specific example, a user inputs a photo of the inside of a factory into the system, and the device sends the photo to the server. The server analyzes the photo using OpenCV and generates a 3D model using Blender. The user then inputs an instruction to the device, such as "move machine A two meters to the right," and the device sends the instruction to the server. The server analyzes the instruction using OpenAI GPT-4 and corrects the 3D model. Furthermore, when the user smiles, the device captures their facial expression and sends it to the server. The server analyzes the user's emotions using the Microsoft Azure Emotion API and adjusts the 3D metaverse environment to create a bright and cheerful atmosphere.

[0430] An example of a prompt sentence might be:

[0431] "Input a photo of the interior of a factory and generate a 3D model. Then move equipment A two meters to the right and change the wall color to blue. Also, if the user is experiencing joy, adjust the 3D metaverse environment to be brighter and more cheerful."

[0432] In this way, the user, terminal, and server work together to realize each function of the system.

[0433] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0434] Step 1:

[0435] The user inputs photos of the interior and exterior of a factory, products, equipment, etc. into the terminal. The terminal then sends these photos to the server. The input is the photo taken by the user, and the output is the photo data sent to the server. Specifically, the user takes a photo with a smartphone or digital camera and uploads it through the terminal's application.

[0436] Step 2:

[0437] The server uses image processing software (e.g., "OpenCV") to analyze the received photos. The input is the photo data sent to the server, and the output is the analyzed image data. Specifically, the server performs edge detection and object recognition on the photos to extract the information necessary to generate a 3D model.

[0438] Step 3:

[0439] The server generates a 3D model using 3D modeling software (e.g., Blender) based on the analysis results. The input is the analyzed image data, and the output is the generated 3D model data. Specifically, the server runs a Blender script to build a 3D model based on the information obtained from the photo.

[0440] Step 4:

[0441] The user inputs a text instruction for the generated 3D model. The terminal sends this text instruction to the server. The input is the text instruction entered by the user, and the output is the text data sent to the server. In concrete terms, the user inputs an instruction such as "move device A 2 meters to the right" into the text input field of the terminal.

[0442] Step 5:

[0443] The server uses a generative AI model (e.g., "OpenAI GPT-4") to analyze the text instructions and correct the 3D model. The input is the text data sent to the server, and the output is the corrected 3D model data. Specifically, the server uses GPT-4 to process the text instructions in natural language and re-executes the Blender script to correct the 3D model.

[0444] Step 6:

[0445] The user expresses emotions to the system (e.g., facial expressions and voice). The device captures the user's facial expressions and voice and sends them to the server. The input is the user's facial expression and voice data, and the output is the emotion data sent to the server. Specifically, the device uses a camera and microphone to record the user's facial expressions and voice.

[0446] Step 7:

[0447] The server analyzes the user's emotions using an emotion recognition engine (e.g., "Microsoft Azure Emotion API"). The input is the emotion data sent to the server, and the output is the analyzed emotion information. Specifically, the server calls the Emotion API to analyze the user's emotions.

[0448] Step 8:

[0449] The server adjusts the 3D metaverse environment based on the analysis results. The input is the analyzed emotion information, and the output is the adjusted 3D metaverse environment data. Specifically, the server re-executes the Blender script and changes the color tone and layout of the environment according to the user's emotion.

[0450] In this way, the user, terminal, and server work together to realize each function of the system.

[0451] (Application example 1)

[0452] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0453] Conventional 3D metaverse environment construction systems require programming and advanced skills, making them difficult for average users to use. They also lack the accuracy required to generate 3D models from photographs and the ability to adjust the environment based on the user's emotions, limiting the user experience. Furthermore, there is a need for a system that can effectively utilize smartphones and head-mounted displays when adjusting the design and atmosphere of virtual stores.

[0454] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes means for building an elaborate 3D metaverse environment without requiring programming or skills; means for creating a 3D model from a photograph; means for making modifications via text instructions; means for inputting equipment performance (load capacity, speed, etc.); means for generating entire factories, warehouses, stores, etc. in a virtual space; means for simulating the operation of production lines and equipment and the flow of people; means for deriving optimal equipment performance, placement, and layout; means for recognizing user emotions and adjusting the 3D metaverse environment based on those emotions; and means for adjusting the design and atmosphere of a virtual store through an application installed on a smartphone or head-mounted display. This allows even ordinary users to easily build an elaborate 3D metaverse environment and adjust the environment based on the user's emotions. Furthermore, the design and atmosphere of a virtual store can be effectively adjusted using a smartphone or head-mounted display.

[0455] "A means of building an elaborate 3D metaverse environment without programming or skills" is a method that allows users to create detailed and realistic 3D virtual spaces without specialized programming knowledge or advanced technology.

[0456] "Means for creating 3D models from photographs" refers to a technology that automatically generates three-dimensional models based on photographs taken by the user.

[0457] The "means for modifying by text instructions" refers to a method for changing attributes such as the position and color of a 3D model based on text instructions entered by the user.

[0458] "Means for inputting equipment performance (load capacity, speed, etc.)" refers to a method for inputting the physical characteristics and operating characteristics of the generated 3D model.

[0459] "Means for generating entire factories, warehouses, stores, etc. in virtual space" refers to technology that reproduces the interior and exterior of actual factories, warehouses, and stores in virtual space.

[0460] "Means for simulating the operation of production lines and equipment, and the flow of people" refers to a method for simulating the operation of production lines, the movement of equipment, and even the flow of people in a virtual space.

[0461] "Means for deriving optimal equipment performance, placement, layout, etc." refers to a method for determining the most efficient equipment performance, placement, and layout based on simulation results.

[0462] "Means of recognizing the user's emotions and adjusting the 3D metaverse environment based on those emotions" refers to technology that reads emotions from the user's facial expressions and behavior, and changes the atmosphere and settings of the virtual space according to those emotions.

[0463] "Means for adjusting the design and atmosphere of a virtual store through an application installed on a smartphone or head-mounted display" refers to a method for changing the design and atmosphere of a virtual store using an application installed on a smartphone or head-mounted display.

[0464] A system for implementing this invention has the following configuration: First, a user uses a smartphone or head-mounted display to take a photo of a factory, warehouse, store, etc. The photo is sent to a server, which then automatically generates a 3D model from the photo.

[0465] Next, the user can modify the generated 3D model by entering text instructions, such as "move device A 2 meters to the right" or "change the wall color to blue," to change the position or color of the 3D model.

[0466] Additionally, the server uses emotion recognition software to recognize the user's emotions. It reads emotions from the user's facial expressions and behavior and adjusts the 3D metaverse environment based on those emotions. For example, if the user is happy, it will change the atmosphere of the virtual space to be bright and cheerful.

[0467] Users can also adjust the design and atmosphere of the virtual store through an application installed on a smartphone or head-mounted display, allowing them to change the design of the virtual store in real time to create the optimal layout and atmosphere.

[0468] For example, a user can input a photo of a store and generate a 3D model. Then, by inputting the instruction "make the wall color blue," the system changes the color of the walls in the 3D model. Furthermore, if the user is happy, the system adjusts the atmosphere of the virtual store to be bright and cheerful.

[0469] An example of a prompt is as follows:

[0470] "Please input a photo of the store taken by the user and generate a 3D model. Next, please input the instruction 'make the wall color blue' and change the wall color of the 3D model. Finally, if the user is happy, please adjust the atmosphere of the virtual store to be bright and fun."

[0471] The system uses OpenCV to process photos, a 3D modeling library to generate 3D models, an emotion recognition library to recognize user emotions, and a metaverse environment adjustment library to adjust the atmosphere of the virtual space, enabling users to easily build elaborate 3D metaverse environments and adjust the environment based on the user's emotions.

[0472] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0473] Step 1:

[0474] A user uses a smartphone or a head-mounted display to take a photo of a factory, warehouse, store, etc. The input is the photograph, and the output is the photo data. This photo data is sent to a server for subsequent processing.

[0475] Step 2:

[0476] The server generates a 3D model based on the received photo data. The input is the photo data, and the output is the generated 3D model. Specifically, it analyzes the photo using OpenCV and automatically generates the 3D model using a 3D modeling library.

[0477] Step 3:

[0478] The user modifies the 3D model by entering text instructions. The input is the user's text instructions, and the output is the modified 3D model. The server then changes the 3D model's attributes, such as its position and color, based on the text instructions.

[0479] Step 4:

[0480] The server uses emotion recognition software to recognize the user's emotions. The input is the user's facial expressions and behavior data, and the output is the recognized emotion information. Specifically, the server uses an emotion recognition library to analyze the user's emotions.

[0481] Step 5:

[0482] The server adjusts the 3D metaverse environment based on the recognized emotional information. The input is emotional information, and the output is the adjusted 3D metaverse environment. Specifically, it changes the atmosphere and settings of the virtual space using a metaverse environment adjustment library.

[0483] Step 6:

[0484] Users adjust the design and atmosphere of the virtual store in real time through an application installed on a smartphone or head-mounted display. The input is the user's design instructions and emotional information, and the output is the adjusted design and atmosphere of the virtual store. Specifically, users operate the application to change the layout and color of the 3D model.

[0485] In this way, users can easily build sophisticated 3D metaverse environments and adjust them based on emotions.

[0486] Example 2

[0487] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0488] Conventional simulation systems have had difficulty providing simulation results that take into account the user's emotional state. Furthermore, when inputting equipment performance data, there was a lack of means to obtain optimal simulation results based on the user's emotions. This meant that it was not possible to simulate optimal production line speeds or equipment operation under stressful conditions.

[0489] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0490] In this invention, the server includes means for constructing an elaborate virtual environment without requiring programming or skills, means for creating a three-dimensional model from an image, means for making corrections using text instructions, means for inputting equipment performance (load capacity, speed, etc.), means for generating an entire facility in a virtual space, means for simulating the operation of production lines and equipment and the flow of people, means for deriving optimal equipment performance, placement, layout, etc., means for acquiring user emotional data, and means for adjusting the simulation results based on the emotional data. This makes it possible to provide optimal simulation results that take into account the user's emotional state.

[0491] "A means of building an elaborate virtual environment without the need for programming or skills" refers to technology that allows even users without specialized programming knowledge or skills to easily create highly accurate virtual environments.

[0492] "Means for creating a 3D model from an image" refers to technology that automatically generates a 3D model based on 2D image data.

[0493] "Means for modifying with text instructions" refers to a technology that allows a user to modify a virtual environment or a three-dimensional model by inputting instructions in text format.

[0494] "Means for inputting equipment performance (load capacity, speed, etc.)" refers to technology that allows users to input specific performance data of equipment (e.g., load capacity and speed) into the system.

[0495] "Means for generating an entire facility in a virtual space" refers to technology that reproduces an entire facility such as a factory, warehouse, or store in a virtual space.

[0496] "Means for simulating the operation of production lines and equipment, and the flow of people" refers to technology that simulates the operation of production lines, equipment, and the flow of people in a virtual space.

[0497] "Means of deriving optimal equipment performance, placement, layout, etc." refers to technology that determines optimal equipment performance, placement, and layout based on simulation results.

[0498] "Means of acquiring user emotional data" refers to technology that analyzes the user's emotional state from facial expressions, voice, etc., and acquires that data.

[0499] "Means for adjusting simulation results based on emotional data" refers to a technology for adjusting simulation results based on acquired emotional data of a user.

[0500] This invention is a system in which a user inputs performance data of a device, simulates the performance using simulation software, and further adjusts the simulation results based on the user's emotional data. A specific embodiment of this system will be described below.

[0501] First, the user uses a terminal to input the equipment's performance data (e.g., load capacity of 500 kg, maximum speed of 1 m / s) into the system. Input is done through a dedicated interface. This interface is designed to allow users to input data easily.

[0502] The server then receives the performance data sent by the user and launches simulation software (e.g., ANSYS or MATLAB®) to simulate the performance of the equipment.

[0503] The server inputs the performance data entered by the user into the launched simulation software. For example, enter data such as "maximum load capacity 500 kg, maximum speed 1 m / s" into ANSYS and run the simulation. While the simulation is running, the server monitors the progress.

[0504] Furthermore, the server obtains the user's current emotional data through the emotion engine, which analyzes the user's stress or relaxation state from their facial expressions and voice. This emotional data is used to reflect the user's emotional state in real time.

[0505] The server adjusts the simulation results based on the acquired emotional data. For example, if the user is feeling stressed, the server generates a simulation result that slows down the production line. This allows the server to provide optimal simulation results that take the user's emotional state into account.

[0506] Finally, the server provides the adjusted simulation results to the user, who can check them through their terminal. As a concrete example, the user inputs the following prompt sentence into the generative AI model:

[0507] Example prompt sentence:

[0508] "The maximum load capacity of equipment A is 500 kg, and its maximum speed is 1 m / s. Please conduct a simulation that takes these performance characteristics into account and show us the results. Also, if the user feels stressed, please provide us with the results of a simulation that reduces the speed of the production line."

[0509] By inputting this prompt sentence, the server uses simulation software to simulate the performance of the device, and uses an emotion engine to provide optimal simulation results according to the user's emotions.

[0510] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0511] Step 1:

[0512] The user inputs the performance data of the device.

[0513] The user opens a dedicated interface on the terminal and inputs "Device A's maximum load capacity is 500 kg, and its maximum speed is 1 m / s." The input data is sent from the terminal to the server. The input data becomes the base data for simulating the device's performance.

[0514] Step 2:

[0515] The server receives the input data and runs the simulation software.

[0516] The server receives the performance data sent by the user. After receiving the data, the server launches the simulation software (e.g., ANSYS or MATLAB). The launched software prepares to run a simulation based on the performance data.

[0517] Step 3:

[0518] The server inputs performance data into the simulation software and executes the simulation.

[0519] The server inputs the performance data entered by the user into the launched simulation software. For example, data such as "maximum load capacity 500 kg, maximum speed 1 m / s" is entered into ANSYS and the simulation is started. As the simulation progresses, the server monitors the progress and obtains the simulation results. As an output, detailed simulation results regarding the operation and performance of the equipment are obtained.

[0520] Step 4:

[0521] The server acquires the user's emotion data from the emotion engine.

[0522] The server obtains the user's current emotional data through the emotion engine. The emotion engine analyzes the user's stress or relaxation state from their facial expressions and voice. The user's facial expressions and voice data are used as input, and the user's emotional state (e.g., stress level) is obtained as output.

[0523] Step 5:

[0524] The server adjusts the simulation results based on the emotion data.

[0525] The server adjusts the simulation results based on the acquired emotional data. For example, if a user is feeling stressed, the server generates a simulation result that slows down the speed of the production line. The emotional data and the simulation results are used as inputs, and the adjusted simulation results are obtained as outputs.

[0526] Step 6:

[0527] The server provides the final simulation results to the user.

[0528] The server generates the adjusted simulation results and sends them to the user's device. The user sees the results on the device screen, which reads, "Device A can withstand a load of 500 kg and can operate at a maximum speed of 1 m / s. However, based on the current stress level, the production line speed will be adjusted to 0.8 m / s." The adjusted simulation results are used as input, and the final simulation results provided to the user are obtained as output.

[0529] (Application example 2)

[0530] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0531] Conventional simulation systems for the operation of production lines and equipment in factories, warehouses, stores, etc. are unable to take into account the emotional state of the user, making it difficult to provide optimal simulation results even in situations where the user is under stress. Furthermore, when feeding back the simulation results to actual business operations, the user's emotional state cannot be reflected, which creates the problem of not reducing user stress in actual operations.

[0532] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0533] In this invention, the server includes means for constructing an elaborate 3D virtual environment without requiring programming or skills, means for creating a 3D model from an image, means for making modifications using text instructions, means for inputting equipment performance (load capacity, speed, etc.), means for generating entire factories, warehouses, stores, etc. in a virtual space, means for simulating the operation of production lines and equipment and the flow of people, means for deriving optimal equipment performance, placement, layout, etc., means for detecting the user's emotional state and adjusting the simulation results, and means for displaying the adjusted simulation results. This makes it possible to provide optimal simulation results that take the user's emotional state into consideration, thereby reducing user stress in actual business operations.

[0534] A "3D virtual environment" is a three-dimensional virtual space generated on a computer, which users can experience visually.

[0535] "Means for creating 3D models from images" refers to a technology that generates three-dimensional solid models based on two-dimensional image data.

[0536] "Means for making modifications based on text instructions" refers to a technology that automatically modifies a 3D model or virtual environment based on text instructions entered by a user.

[0537] "Means for inputting equipment performance" refers to an interface that allows users to input equipment characteristics (load capacity, speed, etc.) into the system.

[0538] "Means of generating in virtual space" refers to technology that reproduces physical spaces such as factories, warehouses, and stores within a virtual environment.

[0539] "Means for simulating the operation of production lines and equipment, and the flow of people" refers to technology that reproduces and simulates the operation of production lines and equipment, and the flow of people, within a virtual environment.

[0540] "Means for deriving optimal equipment performance, placement, layout, etc." refers to technology that proposes optimal equipment performance, placement, and layout based on simulation results.

[0541] The "means for detecting the emotional state of the user and adjusting the simulation results" is a technology for detecting the emotions of the user and adjusting the simulation results based on those emotions.

[0542] The "means for displaying the adjusted simulation results" is a technique for visually displaying to the user the simulation results adjusted based on the emotional state.

[0543] A system for implementing the present invention has the following configuration.

[0544] System Program

[0545] 1. Hardware and Software Use

[0546] Hardware: Smartphone (iOS or Android)

[0547] Software: Python, TensorFlow, Keras, OpenCV, Emotion API (Microsoft Azure, etc.)

[0548] 2. Data Entry

[0549] Users input the factory robot's performance (load capacity, speed, etc.) via a smartphone app.

[0550] Example: "Load capacity: 500kg, Maximum speed: 1m / s"

[0551] 3. Running the simulation

[0552] The server uses Python and TensorFlow to perform simulations based on the input performance data.

[0553] Display simulation results in real time.

[0554] 4. Operation of the Emotion Engine

[0555] Using the smartphone camera, it analyzes the user's face and detects their emotional state using OpenCV and the Emotion API.

[0556] Example: "Users are stressed."

[0557] 5. Adjusting the simulation results

[0558] The server adjusts the simulation results based on the results of the emotion engine.

[0559] Example: "If the user is stressed, the system adjusts the speed of the production line to 0.8 m / s."

[0560] 6. Displaying the results

[0561] The adjusted simulation results are displayed to the user.

[0562] Example: "With the current settings, the robot's speed is 0.8 m / s."

[0563] Program processing explanation

[0564] The server receives factory robot performance data entered by the user through a smartphone app and runs a simulation using Python and TensorFlow. The simulation results are displayed to the user in real time. The server then uses the smartphone's camera to analyze the user's face using OpenCV and the Emotion API to detect their emotional state. Based on this emotional state, the server adjusts the simulation results and displays the adjusted results to the user.

[0565] Examples and prompts

[0566] Specific examples

[0567] Input: "Load capacity: 500kg, Maximum speed: 1m / s"

[0568] Emotional state: "Stress"

[0569] Output: "With the current settings, the robot's speed is 0.8 m / s"

[0570] Prompt Sentence Examples

[0571] The user inputs the performance of the factory robot, for example, "load capacity: 500 kg, maximum speed: 1 m / s." The system then uses the smartphone camera to detect the user's emotional state. For example, if the user is feeling stressed, the system will adjust the production line speed to 0.8 m / s. The adjusted simulation results are then displayed to the user.

[0572] In this way, this system can provide the optimal production environment for users by combining performance simulation of factory robots with emotion adjustment.

[0573] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0574] Step 1:

[0575] The user launches the smartphone app and inputs the factory robot's performance data (load capacity, speed, etc.). The input data is sent from the device to the server. An example input is "load capacity: 500 kg, maximum speed: 1 m / s."

[0576] Step 2:

[0577] The server uses Python and TensorFlow to run a simulation based on the received performance data. Specifically, it receives the performance data as input and simulates the robot's behavior using a generative AI model. The simulation results are generated as output.

[0578] Step 3:

[0579] The server sends the simulation results to the terminal in real time, and the terminal displays the simulation results to the user. An example of the display is "With the current settings, the robot's speed is 1 m / s."

[0580] Step 4:

[0581] The device uses the smartphone camera to capture a picture of the user's face and analyzes the user's emotional state using OpenCV and the Emotion API. The user's face image is used as input, and the emotional state (e.g., stress) is obtained as output.

[0582] Step 5:

[0583] The server adjusts the simulation results based on the results of the emotion engine. Specifically, it receives the emotional state as input and recalculates the simulation results. For example, if the user is feeling stressed, the system adjusts the speed of the production line to 0.8 m / s. The adjusted simulation results are generated as output.

[0584] Step 6:

[0585] The server sends the adjusted simulation results to the terminal, and the terminal displays the adjusted simulation results to the user. An example of the display is "With the current settings, the robot's speed is 0.8 m / s."

[0586] In this way, the system processes and calculates data based on the input data at each step, ultimately providing the user with optimal simulation results.

[0587] Example 3

[0588] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0589] Conventional 3D virtual environment construction systems require programming and specialized skills, making them difficult for general users to use. Furthermore, when simulating production lines, equipment operation, and people flow in virtual space, there is a lack of means to derive optimal equipment performance, placement, and layout. Furthermore, the lack of a function to adjust the virtual environment according to the user's emotions limits the user experience. These issues need to be resolved.

[0590] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0591] In this invention, the server includes a means for constructing an elaborate 3D virtual environment without requiring programming or skills, a means for creating a 3D model from a photograph, a means for making modifications via text instructions, a means for inputting equipment performance (load capacity, speed, etc.), a means for generating an entire facility in a virtual space, a means for simulating the operation of production lines and equipment and the flow of people, a means for deriving optimal equipment performance, placement, layout, etc., and a means for recognizing user emotions and adjusting the 3D virtual environment based on those emotions. This allows even general users to easily construct a 3D virtual environment and derive optimal placement and layout through simulation. Furthermore, adjusting the environment according to the user's emotions can provide a richer user experience.

[0592] A "3D virtual environment" is a three-dimensional virtual space generated by a computer, which can be visually experienced by the user.

[0593] "Method of creating 3D models from photographs" is a technology that generates three-dimensional models based on two-dimensional photographic data.

[0594] "Means for modifying by text instructions" refers to a technique for modifying objects and layouts within a 3D virtual environment based on text instructions entered by a user.

[0595] "Means for inputting equipment performance" refers to a technique for inputting performance data (for example, load capacity and speed) of objects used in a virtual environment.

[0596] "Means for generating entire facilities in virtual space" refers to technology that reproduces physical facilities such as factories, warehouses, and stores in virtual space.

[0597] "Means for simulating the operation of production lines and equipment, and the flow of people" refers to technology that simulates the operation of production lines and equipment, and the flow of people, in a virtual space.

[0598] "Means for deriving optimal equipment performance, placement, layout, etc." refers to technology that proposes optimal equipment performance, placement, and layout based on simulation results.

[0599] "Means for recognizing a user's emotions and adjusting a 3D virtual environment based on those emotions" refers to technology that analyzes a user's emotions and adjusts elements within the 3D virtual environment (such as music or lighting) according to those emotions.

[0600] MODE FOR CARRYING OUT THE INVENTION

[0601] This invention is a system that allows users to build sophisticated 3D virtual environments without the need for programming or specialized skills, and derive optimal placement and layout through simulation. It also provides the ability to adjust the virtual environment according to the user's emotions.

[0602] Virtual space generation and simulation

[0603] The server uses a 3D engine such as Unity or Unreal Engine to generate the entire facility, such as a factory, warehouse, or store, specified by the user, in a virtual space. The user can visualize this virtual space using dedicated terminals and software. For example, the user can input the layout and placement information of the space they want to generate through a GUI on their terminal.

[0604] Users can use simulation software such as AnyLogic or FlexSim to simulate production lines, equipment operation, and people flow. Users set simulation parameters and run the simulation. The server analyzes the simulation results and derives optimal equipment performance, placement, and layout. The analysis results are displayed in the form of dashboards and reports that users can use.

[0605] For example, if a user wants to optimize the layout of a store, they can input the following prompt into the generative AI model:

[0606] "Please suggest the optimal layout based on the results of a store's people flow simulation."

[0607] Coordinating 3D Virtual Environments with Emotion Engines

[0608] The server recognizes the user's emotions using an emotion engine, which is built using a general emotion analysis API, for example. While the user is active in the virtual space, the emotion engine analyzes the user's emotions in real time from their facial expressions and voice.

[0609] The server adjusts the 3D virtual environment based on the recognized emotion. For example, when the user feels happy, the server changes the music in the 3D virtual environment to bright and cheerful. Conversely, when the user feels sad, the server changes the music to quiet and calm.

[0610] For example, a prompt that indicates a user is feeling happy might look like this:

[0611] "Change the music in the 3D virtual environment to something upbeat and fun, as users are feeling the joy."

[0612] In this way, the system provides an optimal virtual environment according to the user's emotions. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0613] Virtual space generation and simulation

[0614] Step 1:

[0615] Users use dedicated terminals and software to request the creation of virtual spaces such as factories, warehouses, and stores. As input, the user enters the layout and placement information of the space they want to create through the GUI on their terminal. As output, this information is sent to the server.

[0616] Step 2:

[0617] The server uses a 3D engine such as Unity or Unreal Engine to generate a virtual space based on information specified by the user. As input, it receives layout information sent by the user and processes the data to create a 3D model. As output, the data of the generated virtual space is sent to the device.

[0618] Step 3:

[0619] The user visually checks the generated virtual space using a dedicated device or software. The input is the virtual space data sent from the server, and the output is the virtual space displayed on the device. Specifically, the user uses a VR headset to move freely within the virtual space and check the layout and placement.

[0620] Step 4:

[0621] Using simulation software such as AnyLogic or FlexSim, users can start simulating the operation of production lines and equipment, as well as the flow of people. Simulation parameters are set as input, and simulation results are generated as output. Specifically, users operate the GUI of the simulation software to input parameters.

[0622] Step 5:

[0623] The server analyzes the simulation results and derives optimal equipment performance, placement, and layout. It receives the simulation results as input and performs analysis as data calculations. The output is an optimized proposal. Specifically, the server executes the analysis algorithm and generates the results in the form of a dashboard or report.

[0624] Step 6:

[0625] The user checks the proposals provided by the server in the form of a dashboard or report. The proposed data sent from the server is received as input, and the proposals are displayed on the terminal as output. Specifically, the user readjusts the layout and placement of the virtual space based on the proposals.

[0626] Coordinating 3D Virtual Environments with Emotion Engines

[0627] Step 1:

[0628] The user begins to act in the virtual space. The input is data from the virtual space, and the output is activity within the virtual space. Specifically, the user puts on a VR headset and explores the virtual space.

[0629] Step 2:

[0630] The server uses an emotion engine to recognize the user's emotions. It receives the user's facial expressions and voice data as input and performs emotion analysis as data processing. The output is the recognized emotion data. Specifically, the server uses an emotion analysis API to analyze emotions in real time.

[0631] Step 3:

[0632] The server adjusts the 3D virtual environment based on the recognized emotions. It receives emotion data as input and performs environmental adjustments as data calculations. The adjusted environmental data is generated as output. Specifically, the server changes the music and lighting in the environment through Unity or Unreal Engine.

[0633] Step 4:

[0634] The user experiences a tailored 3D virtual environment. The tailored environment data is received as input, and the tailored environment is displayed on the device as output. The user continues to work in an optimal environment according to their own emotions.

[0635] (Application example 3)

[0636] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0637] Conventional 3D virtual environment creation systems do not adjust the environment based on the user's emotions, resulting in a poor user experience. Furthermore, simulations that take emotional factors into account were not performed when optimizing store or factory layouts, making it difficult to provide an optimal environment. Furthermore, the system required programming and specialized skills, making it difficult for general users to use.

[0638] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0639] In this invention, the server includes means for constructing an elaborate 3D virtual environment without requiring programming or skills, means for creating a 3D model from an image, means for making modifications using text instructions, means for inputting equipment performance (load capacity, speed, etc.), means for generating an entire factory, warehouse, store, etc. in a virtual space, means for simulating the operation of production lines and equipment and the flow of people, means for deriving optimal equipment performance, placement, layout, etc., means for recognizing a user's emotions and adjusting the virtual environment based on those emotions, and means for capturing the user's emotions using a smart device and adjusting the music and lighting in the virtual environment. This makes it possible to provide an optimal virtual environment according to the user's emotions and optimize the layout of a store or factory by taking emotional factors into consideration.

[0640] "A means of creating sophisticated 3D virtual environments without the need for programming or skills" is a technology that allows users to easily create high-quality 3D virtual spaces without having specialized knowledge or skills.

[0641] "Means for creating 3D models from images" refers to a technology that automatically generates three-dimensional 3D models based on photographs or image data.

[0642] "Means for modifying by text instructions" refers to a technique that allows a user to modify objects and layouts in a 3D virtual environment by inputting instructions in text format.

[0643] "Means for inputting equipment performance (load capacity, speed, etc.)" refers to a technique for inputting performance data of objects and equipment used within a virtual environment.

[0644] "Means for generating entire factories, warehouses, stores, etc. in virtual space" refers to technology that reproduces actual physical spaces such as factories, warehouses, and stores in virtual space.

[0645] "Means for simulating the operation of production lines and equipment, and the flow of people" refers to technology that simulates the operation of production lines and equipment, and the flow of people, in a virtual space.

[0646] "Means for deriving optimal equipment performance, placement, layout, etc." refers to technology that proposes optimal equipment performance, placement, and layout based on simulation results.

[0647] "Means for recognizing a user's emotions and adjusting the virtual environment based on those emotions" refers to technology that detects a user's emotions and adjusts elements in the virtual environment (such as music or lighting) according to those emotions.

[0648] "Means for capturing a user's emotions using a smart device and adjusting music and lighting in a virtual environment" refers to technology that uses a device such as a smartphone or tablet to capture a user's emotions and change the music and lighting in a virtual environment based on those emotions.

[0649] A system for carrying out this invention has the following configuration: The server includes means for constructing an elaborate 3D virtual environment without the need for programming or skills, means for creating a 3D model from an image, means for making modifications using text instructions, means for inputting equipment performance (load capacity, speed, etc.), means for generating entire factories, warehouses, stores, etc. in a virtual space, means for simulating the operation of production lines and equipment and the flow of people, means for deriving optimal equipment performance, placement, layout, etc., means for recognizing user emotions and adjusting the virtual environment based on those emotions, and means for capturing user emotions using a smart device and adjusting the music and lighting in the virtual environment.

[0650] Hardware and software used

[0651] Hardware: smartphones, tablets, cameras, speakers

[0652] Software: OpenCV (image capture), Emotion Recognition API (emotion recognition), Unity (3D virtual environment generation and adjustment)

[0653] Data processing and calculation

[0654] 1. Image capture: When a user is using a smartphone or tablet, the camera captures an image of the user's face.

[0655] 2. Emotion Recognition: The captured image is sent to the emotion recognition API to recognize the user's emotions. The emotion recognition API analyzes the image data and identifies the user's emotions (e.g., joy, sadness, etc.).

[0656] 3. Virtual environment adjustment: Adjust the music and lighting in the Unity environment based on the recognized emotion, for example, change the music and lighting to brighter if the user is happy, or change the music and lighting to quieter if the user is sad.

[0657] Specific examples

[0658] When a user is using their smartphone in a store, the camera captures their face and recognizes their emotions. If the user is happy, the music in the store will change to a bright and cheerful one and the lights will be brighter. Conversely, if the user is sad, the music will change to a quiet and calm one and the lights will be dim.

[0659] Prompt Sentence Examples

[0660] When a user uses a smartphone, capture their face with the camera and recognize their emotions using the emotion recognition API. Based on the recognized emotion, adjust the music and lighting in the Unity environment. For example, if the user is happy, use bright music and lighting, and if the user is sad, use quiet music and subdued lighting.

[0661] In this way, it becomes possible to provide an optimal virtual environment according to the user's emotions and to optimize the layout of stores and factories by taking emotional factors into account.

[0662] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0663] Step 1:

[0664] When a user uses a smartphone, the device's camera captures an image of the user's face.

[0665] Input: Smartphone camera image

[0666] Output: Captured user's face image

[0667] Specific behavior: The smartphone's camera app launches, detects the user's face, and captures an image.

[0668] Step 2:

[0669] The device sends the captured facial image to an emotion recognition API to recognize the user's emotions.

[0670] Input: A captured image of the user's face

[0671] Output: Emotion data returned from the emotion recognition API (e.g., joy, sadness, etc.)

[0672] Specific operation: The device sends the captured image data to the emotion recognition API and receives the emotion data returned by the API.

[0673] Step 3:

[0674] The server adjusts the music and lighting in the virtual environment based on the emotional data received from the emotion recognition API.

[0675] Input: Emotion data from the emotion recognition API

[0676] Output: Adjusted virtual environment settings (music, lighting, etc.)

[0677] Specific operation: The server analyzes the emotional data and changes the settings to bright music and lighting if the user is happy, and to quiet music and subdued lighting if the user is sad.

[0678] Step 4:

[0679] The server transmits the adjusted settings of the virtual environment to the terminal, and the terminal displays the virtual environment based on the settings.

[0680] Input: Adjusted virtual environment settings

[0681] Output: The adjusted virtual environment displayed on the user's smartphone.

[0682] Specific operation: The server sends the adjusted configuration data to the terminal, and the terminal receives the data and updates the virtual environment.

[0683] Step 5:

[0684] The user experiences the adjusted virtual environment and provides feedback as needed.

[0685] Input: Adjusted virtual environment

[0686] Output: User feedback (e.g. satisfied, dissatisfied, etc.)

[0687] How it works: Users experience the calibrated virtual environment on their smartphone and provide feedback and suggestions for improvement using the in-app feedback feature.

[0688] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0689] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0690] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0691] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0692] [Second embodiment]

[0693] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0694] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0695] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0696] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0697] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0698] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0699] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0700] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0701] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0702] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0703] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0704] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0705] "Example 1"

[0706] In one embodiment of the present invention, a user first inputs a photograph into the system. This photograph may be, for example, a photograph of the interior or exterior of a factory, a product, or equipment. The system then automatically generates a 3D model from the photograph. The generated 3D model can be modified based on the user's text instructions. For example, the 3D model can be easily modified by inputting instructions into the system, such as "move equipment A two meters to the right" or "change the wall color to blue."

[0707] "Example 2"

[0708] Next, the user inputs the equipment's performance (load capacity, speed, etc.) into the system. This allows the system to accurately simulate the operation and performance of the equipment. For example, by inputting information such as the maximum load capacity of equipment A being 500 kg and the maximum speed being 1 m / s into the system, it is possible to perform a simulation that takes these performances into account. "Example 3"

[0709] Furthermore, users can generate entire factories, warehouses, stores, etc. in a virtual space, allowing them to visually grasp the overall layout and placement. Furthermore, by simulating the operation of production lines and equipment, as well as people flow, it is possible to derive optimal equipment performance, placement, and layout. For example, the system can automatically generate proposals for optimizing store layouts based on the results of people flow simulations.

[0710] The processing flow of each embodiment will be described below.

[0711] "Example 1"

[0712] Step 1: The user enters photos of the factory interior, exterior, products, equipment, etc. into the system.

[0713] Step 2: The system automatically generates a 3D model from the input photo.

[0714] Step 3: The user modifies the generated 3D model based on text instructions, such as "move device A 2 meters to the right" or "color the wall blue."

[0715] "Example 2"

[0716] Step 1: The user inputs the equipment's performance (load capacity, speed, etc.) into the system.

[0717] Step 2: The system simulates the operation and performance of the equipment based on the input performance information. For example, input information such as the maximum load capacity of Equipment A being 500 kg and the maximum speed being 1 m / s into the system.

[0718] "Example 3"

[0719] Step 1: The user generates an entire factory, warehouse, store, etc. in a virtual space.

[0720] Step 2: The system simulates the operation of production lines, equipment, and people flow in the generated virtual space.

[0721] Step 3: Based on the simulation results, the system derives optimal equipment performance, placement, layout, etc. For example, it automatically generates proposals for optimizing store layouts based on the results of a people flow simulation.

[0722] Example 1

[0723] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0724] Conventional 3D model generation systems require specialized programming and skills, making them difficult for general users to use. Furthermore, there were limitations on the accuracy of creating 3D models from photographs and the flexibility of modification based on user instructions. Furthermore, there was a lack of a way to feed back simulation results based on the generated 3D models into actual business operations.

[0725] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0726] In this invention, the server includes: means for inputting images captured by a user; means for the server to analyze the input images; means for the server to generate a three-dimensional model based on the analyzed data; means for the user to input text instructions; means for the server to analyze the text instructions and modify the three-dimensional model; means for inputting equipment performance; means for generating an entire factory, warehouse, store, etc. in a virtual space; means for simulating the operation of production lines and equipment and the flow of people; and means for deriving optimal equipment performance, placement, layout, etc. This allows users without specialized skills to easily generate and flexibly modify highly accurate three-dimensional models. Furthermore, the simulation results can be fed back into actual business operations.

[0727] A "user" is an individual or corporation that uses the system to input images and generate or modify three-dimensional models.

[0728] A "server" is a computer system that analyzes images input by a user and generates and modifies three-dimensional models.

[0729] "Images" refer to photographs or images taken by users and entered into the system.

[0730] "Analysis" means that the server extracts feature points, object contours, colors, etc. from the input image and processes them as data.

[0731] A "three-dimensional model" is a three-dimensional digital representation generated based on analyzed data.

[0732] A "text instruction" is a sentence that describes the details of the corrections or changes that the user will make to the three-dimensional model.

[0733] "Natural language processing" is a technology that allows a server to understand text instructions from a user and make appropriate corrections.

[0734] A "virtual space" is a simulated environment that digitally recreates the entirety of a factory, warehouse, store, etc.

[0735] "Simulation" is a technology that reproduces production lines, equipment operations, people flow, etc. in a virtual space to derive optimal placement and layout.

[0736] "Devices" are digital data such as three-dimensional models and simulation results generated by the system.

[0737] "Performance" refers to the physical characteristics of the equipment, such as load capacity and speed.

[0738] The present invention is a system that generates a three-dimensional model based on an image taken by a user and then modifies the model based on text instructions from the user. A specific embodiment of this system will be described below.

[0739] First, the user uses a terminal to upload images of the interior and exterior of the factory, products, equipment, etc. to the system. These images are in common image formats such as JPEG and PNG. The user opens the terminal's browser, accesses the system's upload page, clicks the "Select File" button to select the image to upload, and clicks the "Upload" button to send the image to the system.

[0740] Next, the server receives the uploaded image and analyzes it using image processing software (e.g., OpenCV or TensorFlow). The server receives the image data and launches the image processing software to analyze the image's feature points, object contours, color, etc.

[0741] Based on the analyzed data, the server automatically generates a 3D model using 3D model generation software (e.g., Blender or Autodesk Maya). The server inputs the analysis results into the 3D model generation software, which then constructs a 3D model based on the image information. The generated 3D model is stored on the server and is ready to be displayed to the user.

[0742] The user checks the generated 3D model in the browser on their device and enters specific instructions for the parts they want to modify in the text boxes. For example, they might say, "Move device A two meters to the right" or "Change the wall color to blue." When the user clicks the "Submit" button, the instructions are sent to the system.

[0743] The server analyzes the text instructions from the user using natural language processing software (e.g., GPT-4 or BERT) to understand the content of the instructions. The server receives the text instructions, launches the natural language processing software, analyzes the text instructions, and extracts specific corrections.

[0744] Based on the understood instructions, the server again uses the 3D model generation software to modify the 3D model. The server inputs the analysis results into the 3D model generation software, which modifies the 3D model based on the instructions. The modified 3D model is stored on the server and displayed to the user.

[0745] For example, if a user uploads an image of the inside of a factory to the system, the server receives the image, analyzes it using OpenCV, and generates a 3D model using Blender. Next, if the user inputs the instruction "move equipment A 2 meters to the right," the server analyzes this instruction using GPT-4 and again uses Blender to move equipment A in the 3D model 2 meters to the right.

[0746] An example of a prompt is as follows:

[0747] "Upload a photo of the interior of your factory. Then, if you want to make any modifications to the generated 3D model, enter specific instructions in text. For example, 'Move equipment A two meters to the right,' or 'Change the color of the walls to blue.'"

[0748] The above is a specific embodiment for carrying out the present invention.

[0749] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0750] Step 1:

[0751] A user uploads an image using a device.

[0752] Specific operation: The user opens the browser on the device, accesses the system's upload page, clicks the "Select file" button, selects the image to upload, and clicks the "Upload" button.

[0753] Input: User-taken image (JPEG or PNG format)

[0754] Output: Image data sent to the server

[0755] Step 2:

[0756] The server receives the uploaded images and analyzes them using image processing software.

[0757] How it works: The server receives the image data and launches image processing software such as OpenCV or TensorFlow, which analyzes the image's features, object contours, color, etc.

[0758] Input: Uploaded image data

[0759] Output: Analyzed image data (information such as feature points, contours, and colors)

[0760] Step 3:

[0761] The server generates a three-dimensional model based on the analyzed data using three-dimensional model generation software.

[0762] How it works: The server inputs the analysis results into 3D model generation software such as Blender or Autodesk Maya, which then creates a 3D model based on the image information.

[0763] Input: Analyzed image data

[0764] Output: Generated 3D model

[0765] Step 4:

[0766] The user uses a terminal to check the generated three-dimensional model and inputs correction instructions in text.

[0767] Specific operation: The user checks the 3D model in the browser on their device, enters specific instructions about the parts they want to correct in the text box, and clicks the "Submit" button.

[0768] Input: Generated 3D model, user text instructions

[0769] Output: Text instructions sent to the server

[0770] Step 5:

[0771] The server analyzes the text instructions from the user using natural language processing software.

[0772] Specific operation: The server receives the text instructions and launches natural language processing software such as GPT-4 or BERT. The software analyzes the text instructions and extracts specific corrections.

[0773] Input: User's text instructions

[0774] Output: Parsed modifications

[0775] Step 6:

[0776] The three-dimensional model generation software is then used again to modify the three-dimensional model based on the instructions understood by the server.

[0777] Specific operation: The server inputs the analysis results into the 3D model generation software. The software modifies the 3D model based on the instructions. The server saves the modified 3D model and displays it to the user.

[0778] Input: Analyzed corrections, generated 3D model

[0779] Output: Modified 3D model

[0780] The above is the specific flow of the program processing of this system.

[0781] (Application example 1)

[0782] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0783] Conventionally, optimizing the layout and equipment placement of factories, warehouses, stores, etc. required specialized knowledge and skills, and was problematic in that it was time-consuming and costly. Furthermore, there was no system that could generate a 3D model from on-site photos and modify it based on user instructions. This made it difficult to create a virtual environment that accurately reflected on-site conditions and perform efficient simulations.

[0784] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0785] In this invention, the server includes means for constructing an elaborate 3D virtual environment without requiring programming or skills, means for creating a 3D model from a photograph, means for making modifications based on text instructions, means for inputting equipment performance (load capacity, speed, etc.), means for generating entire factories, warehouses, stores, etc. in a virtual space, means for simulating the operation of production lines and equipment and the flow of people, means for deriving optimal equipment performance, placement, layout, etc., means for taking photographs of the inside of a factory using a smartphone and generating a 3D model from the photograph, means for modifying the generated 3D model based on the user's text instructions, and means for displaying the modified 3D model. This allows users to generate 3D models based on photos of the site and easily modify them, enabling efficient optimization of layout and equipment placement.

[0786] "A means of creating an elaborate 3D virtual environment without the need for programming or skills" is a method that allows even users without specialized knowledge or skills to easily create a highly accurate 3D virtual space.

[0787] "Means for creating 3D models from photographs" refers to a technology that automatically generates 3D models based on photographs taken by users.

[0788] The "means for modifying by text instructions" is a method for modifying a 3D model based on text instructions entered by a user.

[0789] "Means for inputting equipment performance (load capacity, speed, etc.)" refers to a method for inputting performance and characteristics into the generated 3D model.

[0790] "Means for generating entire factories, warehouses, stores, etc. in virtual space" refers to technology that reproduces actual spaces such as factories, warehouses, and stores in virtual space.

[0791] "Means for simulating the operation of production lines and equipment, and the flow of people" refers to a method for simulating the operation of production lines and equipment, and the flow of people in a virtual space.

[0792] "Means for deriving optimal equipment performance, placement, layout, etc." refers to a method for determining optimal equipment performance, placement, and layout based on simulation results.

[0793] "A means of taking photos of the inside of a factory using a smartphone and generating a 3D model from those photos" is a technology that generates a 3D model based on photos of the inside of a factory taken with a smartphone.

[0794] The "means for modifying the generated 3D model based on the user's text instructions" is a method for modifying the generated 3D model according to the text instructions entered by the user.

[0795] The "means for displaying the modified 3D model" is a technology for visually displaying the modified 3D model.

[0796] A system for implementing this invention is configured as follows: First, a user takes a photo of the inside of a factory using a smartphone. The taken photo is input into the system, which automatically generates a 3D model from the photo. The generated 3D model can be modified based on the user's text instructions. For example, possible instructions include "move machine A 3 meters to the left" or "change the color of the wall to green." The modified 3D model is visually displayed, allowing the user to confirm the results.

[0797] This system is realized using the following hardware and software: a smartphone, a server, and a display device. The software uses OpenCV, Blender, and OpenAI's GPT-3.

[0798] The specific process is as follows: First, a photo taken with a smartphone is sent to the server. The server analyzes the photo using OpenCV and generates a 3D model. The generated 3D model is visually displayed using Blender. When the user enters text instructions, the instructions are sent to the server and analyzed using OpenAI's GPT-3. Based on the analysis results, the 3D model is modified and displayed again in Blender.

[0799] As a concrete example, consider a scenario in which the layout of machines in a factory needs to be changed. A user takes a photo of the factory and inputs the instruction "move machine A 3 meters to the left." The system generates a 3D model from the photo and repositions machine A based on the instruction. The modified 3D model is visually displayed to the user, allowing the user to confirm the results.

[0800] An example of a prompt is as follows:

[0801] "Take a photo of your factory and enter it into the app. Then enter instructions like this:

[0802] "Move machine A 3 meters to the left"

[0803] Change the wall color to green

[0804] "Move product B forward 2 meters."

[0805] In this way, users can easily optimize the layout of their factory.

[0806] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0807] Step 1:

[0808] A user takes a photo of the inside of a factory using a smartphone. The photo is saved in the smartphone. The input is the photo of the inside of the factory, and the output is the image file saved in the smartphone.

[0809] Step 2:

[0810] A user starts the application on their smartphone and uploads a photograph they have taken to the system. The input is the image file on the smartphone, and the output is the image data sent to the server.

[0811] Step 3:

[0812] The server analyzes the received image data using OpenCV and generates a 3D model from the photo. The input is the image data sent to the server, and the output is the generated 3D model data. Specifically, it extracts feature points from the image and reconstructs the 3D shape based on them.

[0813] Step 4:

[0814] The server visually displays the generated 3D model using Blender. The input is the generated 3D model data, and the output is the visually displayed 3D model. Specifically, it uses Blender's API to render the 3D model.

[0815] Step 5:

[0816] The user inputs text instructions through the application, for example, "move machine A 3 meters to the left." The input is the user's text instructions, and the output is the text data sent to the server.

[0817] Step 6:

[0818] The server uses OpenAI's GPT-3 to analyze the received text instructions. The input is the text data sent to the server, and the output is the analysis result. Specifically, the text instructions are processed using the GPT-3 API for natural language processing.

[0819] Step 7:

[0820] The server modifies the 3D model based on the analysis results. The input is the analysis results and the generated 3D model data, and the output is the modified 3D model data. Specifically, it uses Blender's API to change the position and color of the 3D model.

[0821] Step 8:

[0822] The server visually displays the modified 3D model again using Blender. The input is the modified 3D model data, and the output is the visually displayed modified 3D model. Specifically, the server renders the modified 3D model using Blender's API.

[0823] Step 9:

[0824] The user checks the corrected 3D model through the application. The input is the visually displayed corrected 3D model, and the output is the user's confirmation result. Specifically, the corrected 3D model is displayed on the smartphone display.

[0825] Example 2

[0826] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0827] In conventional systems, advanced programming skills were required to build an elaborate virtual environment, and complex settings were often required to run simulations based on equipment performance data. This made it difficult for users to easily build a virtual environment and simulate equipment performance. Furthermore, there was a lack of a way to feed back the simulation results into actual business operations.

[0828] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0829] In this invention, the server includes means for receiving information on the performance of equipment moving between factories, warehouses, and stores, means for generating a virtual space representing the factories, warehouses, and stores, a prompt statement for instructing a simulation of the movement of the equipment in the virtual space based on the performance data of the equipment, means for executing a simulation of the movement of the equipment using a generative AI model, and means for providing a user with the simulation results. This enables a user without advanced skills to build a virtual environment, simulate the performance of equipment, and reflect the results in actual business operations. The generating means in the server generates a three-dimensional model representing the factory, warehouse, and store in a virtual space from photographs representing the factory, warehouse, and store. The executing means in the server executes a simulation of the movement of the equipment using the prompt sentence, the generating AI model, and the simulation software.

[0830] A "virtual environment" is a digital space generated on a computer that mimics the real world.

[0831] A "three-dimensional model" is digital data that represents the shape and structure of an object in three-dimensional space.

[0832] A "text instruction" is an instruction for an operation or correction given to the system by the user using text information.

[0833] "Equipment performance data" refers to data that indicates the physical characteristics, such as the load capacity and speed, of equipment moving between factories, warehouses, and stores.

[0834] A "generative AI model" is a model that uses artificial intelligence technology to set up simulations based on equipment performance data.

[0835] A "prompt" is a text sentence that instructs the generative AI model on the simulation settings and conditions.

[0836] "Simulation" is the process of simulating the operation and performance of equipment in a virtual environment.

[0837] "Feedback" is the process of reflecting the results of the simulation in actual business operations.

[0838] This invention is a system in which a user inputs performance data of a device, a server uses a generated AI model to perform a simulation based on that data, and provides the results to the user. A specific embodiment of this system will be described below.

[0839] Hardware and software used

[0840] Hardware: Servers, user terminals

[0841] Software: Generative AI models, simulation software

[0842] System processing flow

[0843] First, three-dimensional models representing the factories, warehouses, and stores are generated in a virtual space from photographs of the factories, warehouses, and stores in the same manner as described in the first embodiment. Then, the following processing is performed. 1. The user inputs the device performance data.

[0844] The user inputs the performance data of the device into the input form on the terminal. For example, the maximum load capacity of device A is 500 kg, and the maximum speed is 1 m / s.

[0845] 2. The device sends the entered performance data to the server.

[0846] The terminal converts the performance data entered by the user into a data format such as JSON and sends an HTTP request to the server.

[0847] 3. The server receives the performance data and generates a prompt for the generative AI model.

[0848] The server analyzes the received performance data and generates a prompt to input to the generative AI model. For example, it generates a prompt such as, "Please perform an operation simulation when the maximum load capacity of device A is 500 kg and the maximum speed is 1 m / s."

[0849] 4. The server uses the generated AI model to set up the simulation.

[0850] The server inputs a prompt statement into the generative AI model to generate the settings necessary for the simulation. The generative AI model then sets the simulation parameters and conditions based on the prompt statement.

[0851] 5. The server runs the simulation using the simulation software.

[0852] The server launches the simulation software and runs a simulation based on the settings obtained from the generative AI model. The simulation software calculates the behavior and performance of the equipment and generates simulation results.

[0853] 6. The server sends the simulation results to the device.

[0854] The server converts the simulation results into a data format such as JSON and sends them to the terminal as an HTTP response.

[0855] 7. The device displays the simulation results to the user.

[0856] The device analyzes the received simulation results and displays them in a format that is easy for the user to view. For example, the simulation results may show, in graphs and text, how device A behaves when it supports a load of 500 kg or when it operates at a speed of 1 m / s.

[0857] Specific examples

[0858] Prompt Sentence Examples

[0859] "Please perform a simulation of the operation of device A when the maximum load capacity is 500 kg and the maximum speed is 1 m / s."

[0860] Simulation result display example

[0861] "Device A can support a load of 500 kg and has been confirmed to operate stably at a speed of 1 m / s."

[0862] In this way, the server uses the generated AI model and simulation software to simulate the operation and performance of equipment based on performance data entered by the user, and provides the results to the user.This system enables users to create a virtual environment, simulate equipment performance, and reflect the results in actual business operations without requiring advanced skills.

[0863] The flow of the identification process in the second embodiment will be described with reference to FIG. First, a three-dimensional model representing the factory, warehouse, and store is generated in a virtual space from photographs of the factory, warehouse, and store. Then, the following steps are performed.

[0864] Step 1:

[0865] The user inputs the performance data of the device.

[0866] Input: The user enters the equipment's performance data (e.g., maximum load capacity of 500 kg, maximum speed of 1 m / s) into the terminal's input form.

[0867] Specific operation: The user uses the keyboard to enter device performance data into a form displayed on the terminal screen.

[0868] Step 2:

[0869] The terminal transmits the input performance data to the server.

[0870] Input: User-entered performance data.

[0871] Output: JSON formatted data sent to the server.

[0872] Specific operation: The terminal converts the performance data entered by the user into JSON format, creates an HTTP request, and sends it to the server.

[0873] Step 3:

[0874] The server receives the performance data and generates prompt sentences for the generative AI model.

[0875] Input: Performance data sent from the device in JSON format.

[0876] Output: A prompt to be fed into the generative AI model.

[0877] Specific operation: The server analyzes the received performance data and generates a prompt message saying, "Please perform an operation simulation when the maximum load capacity of device A is 500 kg and the maximum speed is 1 m / s."

[0878] Step 4:

[0879] The server uses the generated AI model to set up the simulation.

[0880] Input: The generated prompt statement.

[0881] Output: The configuration parameters required for the simulation.

[0882] Specific operation: The server inputs a prompt statement into the generative AI model, and the generative AI model sets the simulation parameters and conditions based on the prompt statement.

[0883] Step 5:

[0884] The server runs the simulation using the simulation software.

[0885] Input: Simulation setup parameters obtained from the generative AI model.

[0886] Output: Simulation results.

[0887] Specific operation: The server starts the simulation software, inputs the setting parameters, and runs the simulation. The simulation software calculates the behavior and performance of the equipment and generates the simulation results.

[0888] Step 6:

[0889] The server transmits the simulation results to the terminal.

[0890] Input: Simulation results.

[0891] Output: Simulation result data in JSON format sent to the terminal.

[0892] Specific operation: The server converts the simulation results into JSON format and sends them to the terminal as an HTTP response.

[0893] Step 7:

[0894] The terminal displays the simulation results to the user.

[0895] Input: Simulation result data in JSON format sent from the server.

[0896] Output: Simulation results that are displayed to the user.

[0897] Specific operation: The device analyzes the received simulation results and displays them in a format that is easy for the user to view. For example, the simulation results may show the behavior of device A when it supports a load of 500 kg or when it operates at a speed of 1 m / s, as graphs or text.

[0898] (Application example 2)

[0899] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0900] With conventional systems, building virtual environments for factories, warehouses, stores, etc. required programming and specialized skills, making them difficult for general users to use. Furthermore, there was a lack of means to accurately simulate equipment performance, making it difficult to derive optimal equipment performance and placement for actual operation. Furthermore, there was no way to visually check the simulation results, making it difficult for users to intuitively understand the results.

[0901] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0902] In this invention, the server includes means for constructing an elaborate 3D virtual environment without requiring programming or skills, means for creating a 3D model from an image, means for making modifications using text instructions, means for inputting equipment performance (load capacity, speed, etc.), means for generating entire factories, warehouses, stores, etc. in a virtual space, means for simulating the operation and flow of production lines and equipment, means for deriving optimal equipment performance, placement, layout, etc., and means for simulating and visually displaying the operation and performance of robots based on performance data input by the user. This enables users to easily construct a virtual environment without specialized skills, accurately simulate equipment performance, and visually confirm it.

[0903] A "3D virtual environment" is a three-dimensional virtual space generated on a computer, which users can experience visually.

[0904] "Means for creating 3D models from images" refers to a technology that generates three-dimensional solid models based on two-dimensional image data.

[0905] "Means for making corrections based on text instructions" is a technology that automatically makes corrections to the virtual environment or model based on text information entered by the user.

[0906] The "means for inputting equipment performance" is an interface that allows the user to input equipment characteristics (such as load capacity and speed) into the system.

[0907] "Means of generating in virtual space" refers to technology that reproduces real-world environments such as factories, warehouses, and stores in virtual space.

[0908] "Means of simulation" refers to technology that calculates behavior and performance within a virtual environment and mimics actual behavior.

[0909] "Means for deriving optimal equipment performance, placement, layout, etc." refers to technology that calculates the most efficient equipment performance, placement, and layout based on simulation results.

[0910] "Visual display means" refers to technology that graphically displays simulation results, allowing users to intuitively understand them.

[0911] This invention is a system that allows users without specialized skills to easily create a 3D virtual environment, accurately simulate the performance of equipment, and visually check it. A specific embodiment of this system is described below.

[0912] System configuration

[0913] The system consists of the following main components:

[0914] 1. Server: Manages the entire system and processes data.

[0915] 2. User terminal: Provides an interface for user operation.

[0916] 3. Software: Programs for building, simulating and visualizing 3D virtual environments.

[0917] Hardware and software used

[0918] Hardware: Regular PC or tablet

[0919] Software: Python, matplotlib, numpy

[0920] Program processing explanation

[0921] The server receives performance data (such as load capacity and speed) entered by the user and performs a simulation based on this data. Specifically, the process proceeds as follows:

[0922] 1. User input: The user inputs the device's performance data through a terminal, such as "maximum load capacity 500 kg, maximum speed 1 m / s."

[0923] 2. Data processing: The server performs a simulation based on the received performance data. It uses the Python numpy library to calculate the change in speed and load over time.

[0924] 3. Visualization: The simulation results are displayed graphically using the matplotlib library, which allows users to intuitively understand the results.

[0925] Specific examples

[0926] For example, if a user wants to simulate a robot with a maximum load capacity of 500 kg and a maximum speed of 1 m / s, they would enter the following:

[0927] python

[0928] max_load = 500 kg

[0929] max_speed = 1 m / s

[0930] The server runs a simulation based on this data and visually displays the results, allowing users to intuitively understand the robot's behavior and performance.

[0931] Prompt Sentence Examples

[0932] "Please simulate the movement of a robot with a maximum load capacity of 500 kg and a maximum speed of 1 m / s."

[0933] In this way, users can easily create a virtual environment, simulate the performance of equipment, and visually check it, even if they do not have specialized skills.

[0934] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0935] Step 1:

[0936] The user uses a terminal to input the performance data of the equipment (for example, maximum load capacity and maximum speed). The input data is sent to the server. An example of input data is "maximum load capacity 500 kg, maximum speed 1 m / s."

[0937] Step 2:

[0938] The server prepares the simulation based on the received performance data. Specifically, it uses the Python numpy library to generate the time axis data required for the simulation. Based on the input data, time axis data (for example, 100 steps from 0 seconds to 10 seconds) is generated.

[0939] Step 3:

[0940] The server calculates the change in velocity and load based on the generated time axis data. Specifically, it uses the numpy library to calculate the change in velocity and load over time. For example, the velocity is calculated to increase linearly with time, and the load is calculated to increase exponentially with time.

[0941] Step 4:

[0942] The server generates simulation results based on the calculation results. Specifically, it uses the calculated velocity and load data and stores the simulation results in a data structure. The simulation results include velocity and load values ​​at each time step.

[0943] Step 5:

[0944] The server generates graphs using the matplotlib library to visualize the simulation results. Specifically, it plots the changes in speed and load against time and creates a graph to visually display them. The generated graph visually shows the changes in speed and load.

[0945] Step 6:

[0946] The server sends the generated graphs to the user's device, where the user can visually check the simulation results, allowing them to intuitively understand the robot's behavior and performance.

[0947] Step 7:

[0948] Based on the visualized simulation results, users can re-enter performance data as needed and run the simulation again, enabling iterative simulations to derive optimal equipment performance and placement.

[0949] Example 3

[0950] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0951] Determining efficient layouts and equipment placements is extremely important in modern physical spaces such as factories, warehouses, and stores. However, achieving this requires advanced specialized knowledge and skills, and repeated trial and error can be time-consuming and costly. Furthermore, there is a lack of means to analyze simulation results and derive optimal placements and layouts, making efficient operation difficult.

[0952] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0953] In this invention, the server includes means for constructing an elaborate 3D virtual environment without requiring programming or skills, means for creating a 3D model from an image, means for making modifications via text instructions, means for inputting equipment performance (load capacity, speed, etc.), means for generating entire factories, warehouses, stores, etc. in a virtual space, means for simulating the operation of production lines and equipment and the flow of people, means for deriving optimal equipment performance, placement, layout, etc., means for using a generative AI model to analyze the simulation results, and means for automatically generating optimization proposals based on the analysis results. This enables users to determine efficient layouts and equipment placements without requiring advanced specialized knowledge or skills.

[0954] A "3D virtual environment" is a three-dimensional virtual space generated on a computer, in which users can visually recreate and manipulate physical space.

[0955] "Means for creating 3D models from images" refers to a technology that generates three-dimensional models based on two-dimensional image data, and is a means that can create three-dimensional objects from photographs or drawings.

[0956] "Means for modifying by text instructions" refers to a technique that allows a user to modify objects and layouts in a virtual environment by inputting instructions in text format.

[0957] "Means for inputting equipment performance" refers to a technique for inputting performance data (for example, load capacity, speed, etc.) of objects and equipment used in a virtual environment.

[0958] "A means of generating entire factories, warehouses, stores, etc. in virtual space" is a technology that reproduces the entire physical space in a three-dimensional virtual environment, allowing users to visually grasp that space.

[0959] "Means for simulating the operation of production lines and equipment, and the flow of people" refers to technology that simulates the operation of production lines, the movement of equipment, and the flow of people within a virtual environment, and is a means that can simulate actual operations.

[0960] "Means for deriving optimal equipment performance, placement, layout, etc." refers to technology that automatically proposes optimal equipment performance, placement, and layout based on simulation results.

[0961] "Means using generative AI models" refers to technology that uses artificial intelligence models to analyze data and generate results, such as means for analyzing simulation results and proposing optimization.

[0962] "Means for automatically generating optimization proposals based on analysis results" refers to technology that automatically generates proposals for optimal placement and layout based on the analysis results obtained by a generative AI model.

[0963] This invention provides a system that allows a user to generate a physical space such as a factory, warehouse, or store in a virtual space and determine an efficient layout and arrangement of equipment. Specific embodiments of this system are described below.

[0964] Hardware and software used

[0965] Hardware: PC with a high-performance graphics card, VR headset

[0966] Software: Unity (game engine), Python (programming language), generative AI models (e.g., OpenAI's GPT-4)

[0967] Virtual space generation

[0968] Users use Unity to recreate physical spaces such as factories, warehouses, and stores in a virtual space. Specifically, they use Unity's editor to design the building structure and internal layout. For example, they can set the placement of walls, floors, and equipment using drag and drop.

[0969] Entering data

[0970] Users use Python scripts to input production line and equipment operation and people flow data. Specifically, people flow data stored in a CSV file is read using Python and passed to Unity. For example, the pandas library is used to read the CSV file and format the data.

[0971] Running the simulation

[0972] The server runs the simulation on Unity. Specifically, it launches Unity's simulation engine and visualizes the operation of the production line and the movement of people in real time based on the data entered by the user. For example, it displays the operation of equipment and the movement of people in 3D models.

[0973] Analysis of simulation results

[0974] The server uses a generative AI model (e.g., GPT-4) to analyze the simulation results. Specifically, the simulation results are saved in a log file and the data is input into the generative AI model. For example, the people flow data during the simulation is saved in text format and analyzed by GPT-4.

[0975] Optimization suggestions

[0976] Based on the analysis results, the server automatically generates proposals for optimal equipment performance, placement, and layout. Specifically, it presents the optimal layout and placement to the user based on the analysis results obtained from the generative AI model. For example, it makes specific suggestions such as, "Since foot traffic is concentrated in this area, we suggest changing the placement of the cash registers."

[0977] Specific examples

[0978] Factory example

[0979] Virtual space generation: Users design the layout of a factory in Unity, for example, by setting up the layout of production lines and equipment.

[0980] Data input: Users input production line operation data in a Python script, for example, by reading operation data from a CSV file.

[0981] Simulation execution: The server runs the simulation in Unity to visualize the operation of the production line, for example by displaying a 3D model of the equipment in operation.

[0982] Analysis of simulation results: The server uses the generative AI model to analyze the simulation results. For example, the server saves the motion data in text format and runs GPT-4 to analyze it.

[0983] Optimization proposal: The server proposes the optimal layout of the production line, for example, making specific proposals such as "By changing the layout of this equipment, efficiency will improve by 10%."

[0984] Store example

[0985] Creating a virtual space: Users design the layout of a store in Unity, for example, by setting the placement of shelves and cash registers.

[0986] Data input: Users input people flow data in a Python script, for example, by reading people flow data from a CSV file.

[0987] Simulation execution: The server runs the simulation in Unity and visualizes the movement of people, for example, displaying a 3D model of how customers move through a store.

[0988] Analysis of simulation results: The server uses the generative AI model to analyze the simulation results. For example, the server saves the people flow data in text format and runs GPT-4 to analyze it.

[0989] Optimization suggestions: The server will suggest optimal product placement, such as "Since there is a high concentration of people in this area, we suggest changing the placement of popular products."

[0990] Prompt Sentence Examples

[0991] Factory Optimization

[0992] To optimize your factory layout, input production line operation data. Based on the simulation results, we will suggest the optimal layout.

[0993] Store optimization

[0994] Enter foot traffic data to optimize your store layout. Based on the simulation results, we will suggest optimal product placement.

[0995] The flow of the identification process in the third embodiment will be described with reference to FIG.

[0996] Step 1:

[0997] Users use Unity to recreate physical spaces such as factories, warehouses, and stores in virtual spaces.

[0998] Input: User-designed building structure and interior layout data.

[0999] How it works: The user opens the Unity editor and drags and drops the placement of walls, floors, and equipment.

[1000] Output: A 3D model reproduced in virtual space.

[1001] Step 2:

[1002] Users use Python scripts to input production line and equipment operation and people flow data.

[1003] Input: Production line and equipment operation data, and people flow data in a CSV file.

[1004] Specific behavior: The user uses the pandas library to read the CSV file, format the data, and pass it to Unity.

[1005] Output: The formatted data passed to Unity.

[1006] Step 3:

[1007] The server runs the simulation on Unity.

[1008] Input: Production line and equipment operation data, and people flow data entered by users.

[1009] How it works: The server runs Unity's simulation engine and visualizes the operation of the production line and the movement of people in real time, for example, displaying 3D models of how equipment operates and how people move.

[1010] Output: Log file of simulation results.

[1011] Step 4:

[1012] The server uses the generative AI model to analyze the simulation results.

[1013] Input: Simulation results log file.

[1014] Specific operation: The server saves the simulation results in text format and inputs the data into a generative AI model. For example, GPT-4 analyzes the people flow data during the simulation.

[1015] Output: Analysis result data.

[1016] Step 5:

[1017] Based on the analysis results, the server automatically generates proposals for optimal equipment performance, placement, and layout.

[1018] Input: Data from the analysis results obtained from the generative AI model.

[1019] Specific operation: Based on the analysis results, the server presents the user with the optimal layout and placement. For example, it makes a specific suggestion such as, "Since there is a high concentration of people in this area, we suggest changing the placement of the cash registers."

[1020] Output: Optimization suggestions.

[1021] (Application example 3)

[1022] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1023] In conventional store operations, it was difficult to optimize layout and product placement, hindering efficient operations. Furthermore, there was a lack of means to grasp the actual flow of people in the store in real time and instantly change the layout. This resulted in poor customer flow, which could lead to lower sales and lower customer satisfaction.

[1024] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes means for constructing an elaborate 3D virtual environment without requiring programming or skills, means for creating a 3D model from a photograph, means for making modifications using text instructions, means for inputting equipment performance (load capacity, speed, etc.), means for generating an entire factory, warehouse, store, etc. in a virtual space, means for simulating the operation of production lines and equipment and people flow, means for deriving optimal equipment performance, placement, layout, etc., means for proposing optimal product placement and layout based on the results of the people flow simulation in the virtual space, and means for checking and changing the store layout in real time using a smartphone, smart glasses, or head-mounted display. This allows for efficient optimization of store layout and product placement, improving customer flow, and increasing sales and customer satisfaction.

[1025] A "3D virtual environment" is a three-dimensional virtual space generated on a computer, which the user can visually grasp.

[1026] "Means for creating 3D models from photographs" refers to a technology that automatically generates three-dimensional models based on photographic data.

[1027] "Means for modifying by text instructions" refers to a technique that allows a user to modify objects and layouts in a virtual environment by inputting instructions in text format.

[1028] "Means for inputting equipment performance" refers to technology for inputting the performance (e.g., load capacity, speed, etc.) of the generated model or object.

[1029] "Means of generating in virtual space" refers to technology that reproduces actual facilities such as factories, warehouses, and stores in virtual space.

[1030] "Means for simulating the operation of production lines and equipment, and the flow of people" refers to technology that simulates the operation of production lines and equipment, and the flow of people, in a virtual space.

[1031] "Means for deriving optimal equipment performance, placement, layout, etc." refers to technology that calculates and proposes optimal equipment performance, placement, and layout based on simulation results.

[1032] "Means for proposing optimal product placement and layout based on the results of people flow simulation" refers to technology that automatically proposes optimal product placement and layout based on the results of people flow simulation in a virtual space.

[1033] "Means for checking and making changes to store layouts in real time using smartphones, smart glasses, and head-mounted displays" refers to technology that uses these devices to check store layouts in a virtual space in real time and make changes as necessary.

[1034] A system for implementing the present invention uses the following hardware and software.

[1035] Hardware

[1036] Smartphones (e.g. iPhone, Android devices)

[1037] Smart glasses (e.g., Google Glass, Microsoft HoloLens)

[1038] Head-mounted displays (e.g., Oculus Rift, HTC Vive)

[1039] software

[1040] Python 3.x

[1041] NumPy (numerical calculation library)

[1042] Matplotlib (data visualization library)

[1043] SciPy (optimized library)

[1044] System Overview

[1045] The server first provides a means for constructing elaborate 3D virtual environments without requiring programming or skills. This allows users to easily generate virtual spaces. Next, a means for creating 3D models from photographs is used to generate 3D models of actual stores and products. Finally, a means for making corrections using text instructions allows users to easily modify objects and layouts in the virtual space.

[1046] By using a means to input equipment performance (load capacity, speed, etc.), performance data for the generated models and objects can be input, which allows for the creation of entire factories, warehouses, stores, etc. in virtual space.

[1047] The server performs simulations in a virtual space using means for simulating production lines, equipment operations, and people flow. Based on the results of this simulation, a means is provided for deriving optimal equipment performance, placement, layout, etc. Furthermore, it includes means for proposing optimal product placement and layout based on the results of people flow simulation in a virtual space.

[1048] Using a smartphone, smart glasses, or head-mounted display, users can check the store layout in real time and make changes as needed, enabling efficient optimization of store layout and product placement, improving customer flow.

[1049] Specific examples

[1050] For example, when optimizing the layout of a supermarket, the system inputs the product layout as the initial layout and performs a people flow simulation. Based on the results, the system proposes the optimal product placement. Users can use their smartphones or smart glasses to check the layout in real time in the virtual space and make changes as necessary.

[1051] Prompt Sentence Examples

[1052] "Enter the initial layout of your supermarket, run a people flow simulation and suggest the optimal product placement."

[1053] In this way, the virtual store layout optimization assistant optimizes the layout of physical stores and supports efficient store operations.

[1054] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1055] Step 1:

[1056] The server receives initial layout data from the user. The user inputs product placement and store layout using a smartphone or PC. This input data becomes the base data for generating the virtual space.

[1057] Step 2:

[1058] The server generates a three-dimensional model based on the user-provided photo data using a means for creating a 3D model from a photo. This process uses image analysis technology to extract three-dimensional shapes from the photo and build a 3D model. The output is a 3D model that can be used in the virtual space.

[1059] Step 3:

[1060] The server uses a means for making corrections based on text instructions to correct objects and layouts in the virtual space based on text instructions entered by the user. The user inputs correction instructions in text format using a smartphone or PC. The server analyzes the instructions and corrects the corresponding parts in the virtual space. The output is the layout of the corrected virtual space.

[1061] Step 4:

[1062] The server inputs performance data provided by the user using a means for inputting equipment performance (load capacity, speed, etc.). This data is used for simulation in the virtual space. The input data includes the physical properties of each object. The output is a model of the virtual space that reflects the performance data.

[1063] Step 5:

[1064] The server uses a means to generate an entire virtual space, such as a factory, warehouse, or store, based on the data provided by the user. This process integrates the initial layout data, 3D models, and performance data to generate the virtual space. The output is a model of the complete virtual space.

[1065] Step 6:

[1066] The server performs simulations in a virtual space using means to simulate the operation of production lines, equipment, and people. In this process, the server simulates the operation of people and equipment in the virtual space based on data provided by the user. The output is the simulation results.

[1067] Step 7:

[1068] The server calculates the optimal placement using a method to derive the optimal equipment performance, placement, and layout based on the simulation results. In this process, the server analyzes the simulation results and proposes the optimal placement. The output is a proposal for an optimized layout.

[1069] Step 8:

[1070] The server uses a method to propose optimal product placement and layout based on the results of the virtual human flow simulation, and makes specific placement proposals. This process analyzes human flow data and calculates optimal product placement. The output is a proposal for optimal product placement.

[1071] Step 9:

[1072] Users use smartphones, smart glasses, or head-mounted displays to view the store layout in real time and make changes as needed. Using these devices, users visually view the layout in the virtual space and direct changes. The output is a layout of the virtual space updated in real time.

[1073] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1074] "Example 1"

[1075] In one embodiment of the present invention, a system incorporating an emotion engine is provided. The system recognizes a user's emotions and adjusts the 3D metaverse environment based on the user's emotions. Specifically, when the user feels joy, the system adjusts the 3D metaverse environment to a bright and cheerful atmosphere. Conversely, when the user feels sad, the system adjusts the 3D metaverse environment to a quiet and calm atmosphere.

[1076] "Example 2"

[1077] The emotion engine also adjusts the simulation results based on the user's emotions. For example, if a user feels stressed, the system will provide simulation results to alleviate the user's emotions, such as slowing down the production line. This allows users to experience the optimal environment according to their emotions.

[1078] "Example 3"

[1079] Furthermore, the emotion engine recognizes the user's emotions and adjusts the 3D metaverse environment based on those emotions. For example, when the user feels happy, the system changes the music in the 3D metaverse environment to a bright and cheerful one. When the user feels sad, the system changes the music in the 3D metaverse environment to a quiet and calm one. This allows the user to experience the optimal environment according to their emotions.

[1080] The processing flow of each embodiment will be described below.

[1081] "Example 1"

[1082] Step 1: The system recognizes the user's emotions using an emotion engine.

[1083] Step 2: Adjust the 3D metaverse environment based on the recognized emotion. Specifically, when the user feels joy, the system adjusts the 3D metaverse environment to a bright and happy atmosphere.

[1084] Step 3: Conversely, when the user feels sad, the system adjusts the 3D metaverse environment to a quiet and calming atmosphere.

[1085] "Example 2"

[1086] Step 1: The system recognizes the user's emotions using an emotion engine.

[1087] Step 2: Adjust the simulation results based on the recognized emotions. Specifically, when the user feels stressed, the system will provide simulation results to alleviate the user's emotions, such as slowing down the production line.

[1088] "Example 3"

[1089] Step 1: The system recognizes the user's emotions using an emotion engine.

[1090] Step 2: Adjust the 3D metaverse environment based on the recognized emotion. Specifically, when the user feels joy, the system changes the music in the 3D metaverse environment to a brighter, more cheerful one.

[1091] Step 3: Also, when the user feels sad, the system changes the music in the 3D Metaverse environment to a quieter, more soothing one.

[1092] Example 1

[1093] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1094] Previous systems for building 3D metaverse environments required programming and advanced skills, making them difficult for average users to use. Other issues included the accuracy of creating 3D models from photographs and the time required to modify the generated 3D models. Furthermore, the system lacked the functionality to adjust the 3D metaverse environment according to the user's emotions, creating a need for an improved user experience.

[1095] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for creating a 3D model from a photograph, a means for making modifications based on text instructions, and a means for recognizing a user's emotions and adjusting the 3D metaverse environment based on the emotions. This allows a user to generate an elaborate 3D model from a photograph and easily modify it based on text instructions without requiring programming or advanced skills. Furthermore, automatically adjusting the 3D metaverse environment according to the user's emotions can provide a more intuitive and comfortable user experience.

[1096] "Means for creating 3D models from photographs" refers to technology that analyzes photographs entered by users and automatically generates 3D models based on those photographs.

[1097] The "means for performing corrections based on text instructions" is a technology for correcting a generated 3D model based on text instructions entered by a user.

[1098] "Means for recognizing the user's emotions and adjusting the 3D Metaverse environment based on those emotions" refers to a technology that analyzes the user's emotions from facial expressions, voice, etc., and automatically adjusts the 3D Metaverse environment based on the results of that analysis.

[1099] "Means for inputting equipment performance" refers to a technology for inputting the performance of the generated 3D model (e.g., load-bearing capacity, speed, etc.) into the system.

[1100] "Means for generating entire factories, warehouses, stores, etc. in virtual space" refers to technology that reproduces physical spaces such as factories, warehouses, and stores in virtual space.

[1101] "Means for simulating the operation of production lines and equipment, and the flow of people" refers to technology that simulates the operation of production lines and equipment, and the flow of people in a virtual space.

[1102] "Means for deriving optimal equipment performance, placement, layout, etc." refers to technology that determines optimal equipment performance, placement, and layout based on simulation results.

[1103] This invention is a system that allows a user to input a photo, generate a 3D model from the photo, and further adjust the 3D Metaverse environment based on text instructions and emotions. Specific embodiments of this system are described below.

[1104] First, the user inputs photos of the interior and exterior of the factory, products, equipment, etc. into the terminal. The terminal then sends these photos to the server. The server analyzes the received photos and automatically generates a 3D model. This analysis and generation uses image processing software (e.g., "OpenCV") and 3D modeling software (e.g., "Blender").

[1105] Next, the user inputs text instructions for the generated 3D model, such as "move device A two meters to the right" or "change the wall color to blue." The device sends the text instructions to the server, which uses a generative AI model (e.g., "OpenAI GPT-4") to analyze the text instructions and modify the 3D model.

[1106] Furthermore, it also includes the ability to recognize the user's emotions and adjust the 3D metaverse environment based on those emotions. The user expresses their emotions to the system (e.g., facial expressions or voice). The device captures the user's facial expressions and voice and sends them to the server. The server analyzes the user's emotions using an emotion recognition engine (e.g., Microsoft Azure Emotion API). Based on the analysis results, the server adjusts the 3D metaverse environment. For example, if the user is feeling happy, the environment will be adjusted to have a bright and cheerful atmosphere.

[1107] As a specific example, a user inputs a photo of the inside of a factory into the system, and the device sends the photo to the server. The server analyzes the photo using OpenCV and generates a 3D model using Blender. The user then inputs an instruction to the device, such as "move machine A two meters to the right," and the device sends the instruction to the server. The server analyzes the instruction using OpenAI GPT-4 and corrects the 3D model. Furthermore, when the user smiles, the device captures their facial expression and sends it to the server. The server analyzes the user's emotions using the Microsoft Azure Emotion API and adjusts the 3D metaverse environment to create a bright and cheerful atmosphere.

[1108] An example of a prompt sentence might be:

[1109] "Input a photo of the interior of a factory and generate a 3D model. Then move equipment A two meters to the right and change the wall color to blue. Also, if the user is experiencing joy, adjust the 3D metaverse environment to be brighter and more cheerful."

[1110] In this way, the user, terminal, and server work together to realize each function of the system.

[1111] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1112] Step 1:

[1113] The user inputs photos of the interior and exterior of a factory, products, equipment, etc. into the terminal. The terminal then sends these photos to the server. The input is the photo taken by the user, and the output is the photo data sent to the server. Specifically, the user takes a photo with a smartphone or digital camera and uploads it through the terminal's application.

[1114] Step 2:

[1115] The server uses image processing software (e.g., "OpenCV") to analyze the received photos. The input is the photo data sent to the server, and the output is the analyzed image data. Specifically, the server performs edge detection and object recognition on the photos to extract the information necessary to generate a 3D model.

[1116] Step 3:

[1117] The server generates a 3D model using 3D modeling software (e.g., Blender) based on the analysis results. The input is the analyzed image data, and the output is the generated 3D model data. Specifically, the server runs a Blender script to build a 3D model based on the information obtained from the photo.

[1118] Step 4:

[1119] The user inputs a text instruction for the generated 3D model. The terminal sends this text instruction to the server. The input is the text instruction entered by the user, and the output is the text data sent to the server. In concrete terms, the user inputs an instruction such as "move device A 2 meters to the right" into the text input field of the terminal.

[1120] Step 5:

[1121] The server uses a generative AI model (e.g., "OpenAI GPT-4") to analyze the text instructions and correct the 3D model. The input is the text data sent to the server, and the output is the corrected 3D model data. Specifically, the server uses GPT-4 to process the text instructions in natural language and re-executes the Blender script to correct the 3D model.

[1122] Step 6:

[1123] The user expresses emotions to the system (e.g., facial expressions and voice). The device captures the user's facial expressions and voice and sends them to the server. The input is the user's facial expression and voice data, and the output is the emotion data sent to the server. Specifically, the device uses a camera and microphone to record the user's facial expressions and voice.

[1124] Step 7:

[1125] The server analyzes the user's emotions using an emotion recognition engine (e.g., "Microsoft Azure Emotion API"). The input is the emotion data sent to the server, and the output is the analyzed emotion information. Specifically, the server calls the Emotion API to analyze the user's emotions.

[1126] Step 8:

[1127] The server adjusts the 3D metaverse environment based on the analysis results. The input is the analyzed emotion information, and the output is the adjusted 3D metaverse environment data. Specifically, the server re-executes the Blender script and changes the color tone and layout of the environment according to the user's emotion.

[1128] In this way, the user, terminal, and server work together to realize each function of the system.

[1129] (Application example 1)

[1130] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1131] Conventional 3D metaverse environment construction systems require programming and advanced skills, making them difficult for average users to use. They also lack the accuracy required to generate 3D models from photographs and the ability to adjust the environment based on the user's emotions, limiting the user experience. Furthermore, there is a need for a system that can effectively utilize smartphones and head-mounted displays when adjusting the design and atmosphere of virtual stores.

[1132] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes means for building an elaborate 3D metaverse environment without requiring programming or skills; means for creating a 3D model from a photograph; means for making modifications via text instructions; means for inputting equipment performance (load capacity, speed, etc.); means for generating entire factories, warehouses, stores, etc. in a virtual space; means for simulating the operation of production lines and equipment and the flow of people; means for deriving optimal equipment performance, placement, and layout; means for recognizing user emotions and adjusting the 3D metaverse environment based on those emotions; and means for adjusting the design and atmosphere of a virtual store through an application installed on a smartphone or head-mounted display. This allows even ordinary users to easily build an elaborate 3D metaverse environment and adjust the environment based on the user's emotions. Furthermore, the design and atmosphere of a virtual store can be effectively adjusted using a smartphone or head-mounted display.

[1133] "A means of building an elaborate 3D metaverse environment without programming or skills" is a method that allows users to create detailed and realistic 3D virtual spaces without specialized programming knowledge or advanced technology.

[1134] "Means for creating 3D models from photographs" refers to a technology that automatically generates three-dimensional models based on photographs taken by the user.

[1135] The "means for modifying by text instructions" refers to a method for changing attributes such as the position and color of a 3D model based on text instructions entered by the user.

[1136] "Means for inputting equipment performance (load capacity, speed, etc.)" refers to a method for inputting the physical characteristics and operating characteristics of the generated 3D model.

[1137] "Means for generating entire factories, warehouses, stores, etc. in virtual space" refers to technology that reproduces the interior and exterior of actual factories, warehouses, and stores in virtual space.

[1138] "Means for simulating the operation of production lines and equipment, and the flow of people" refers to a method for simulating the operation of production lines, the movement of equipment, and even the flow of people in a virtual space.

[1139] "Means for deriving optimal equipment performance, placement, layout, etc." refers to a method for determining the most efficient equipment performance, placement, and layout based on simulation results.

[1140] "Means of recognizing the user's emotions and adjusting the 3D metaverse environment based on those emotions" refers to technology that reads emotions from the user's facial expressions and behavior, and changes the atmosphere and settings of the virtual space according to those emotions.

[1141] "Means for adjusting the design and atmosphere of a virtual store through an application installed on a smartphone or head-mounted display" refers to a method for changing the design and atmosphere of a virtual store using an application installed on a smartphone or head-mounted display.

[1142] A system for implementing this invention has the following configuration: First, a user uses a smartphone or head-mounted display to take a photo of a factory, warehouse, store, etc. The photo is sent to a server, which then automatically generates a 3D model from the photo.

[1143] Next, the user can modify the generated 3D model by entering text instructions, such as "move device A 2 meters to the right" or "change the wall color to blue," to change the position or color of the 3D model.

[1144] Additionally, the server uses emotion recognition software to recognize the user's emotions. It reads emotions from the user's facial expressions and behavior and adjusts the 3D metaverse environment based on those emotions. For example, if the user is happy, it will change the atmosphere of the virtual space to be bright and cheerful.

[1145] Users can also adjust the design and atmosphere of the virtual store through an application installed on a smartphone or head-mounted display, allowing them to change the design of the virtual store in real time to create the optimal layout and atmosphere.

[1146] For example, a user can input a photo of a store and generate a 3D model. Then, by inputting the instruction "make the wall color blue," the system changes the color of the walls in the 3D model. Furthermore, if the user is happy, the system adjusts the atmosphere of the virtual store to be bright and cheerful.

[1147] An example of a prompt is as follows:

[1148] "Please input a photo of the store taken by the user and generate a 3D model. Next, please input the instruction 'make the wall color blue' and change the wall color of the 3D model. Finally, if the user is happy, please adjust the atmosphere of the virtual store to be bright and fun."

[1149] The system uses OpenCV to process photos, a 3D modeling library to generate 3D models, an emotion recognition library to recognize user emotions, and a metaverse environment adjustment library to adjust the atmosphere of the virtual space, enabling users to easily build elaborate 3D metaverse environments and adjust the environment based on the user's emotions.

[1150] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1151] Step 1:

[1152] A user uses a smartphone or a head-mounted display to take a photo of a factory, warehouse, store, etc. The input is the photograph, and the output is the photo data. This photo data is sent to a server for subsequent processing.

[1153] Step 2:

[1154] The server generates a 3D model based on the received photo data. The input is the photo data, and the output is the generated 3D model. Specifically, it analyzes the photo using OpenCV and automatically generates the 3D model using a 3D modeling library.

[1155] Step 3:

[1156] The user modifies the 3D model by entering text instructions. The input is the user's text instructions, and the output is the modified 3D model. The server then changes the 3D model's attributes, such as its position and color, based on the text instructions.

[1157] Step 4:

[1158] The server uses emotion recognition software to recognize the user's emotions. The input is the user's facial expressions and behavior data, and the output is the recognized emotion information. Specifically, the server uses an emotion recognition library to analyze the user's emotions.

[1159] Step 5:

[1160] The server adjusts the 3D metaverse environment based on the recognized emotional information. The input is emotional information, and the output is the adjusted 3D metaverse environment. Specifically, it changes the atmosphere and settings of the virtual space using a metaverse environment adjustment library.

[1161] Step 6:

[1162] Users adjust the design and atmosphere of the virtual store in real time through an application installed on a smartphone or head-mounted display. The input is the user's design instructions and emotional information, and the output is the adjusted design and atmosphere of the virtual store. Specifically, users operate the application to change the layout and color of the 3D model.

[1163] In this way, users can easily build sophisticated 3D metaverse environments and adjust them based on emotions.

[1164] Example 2

[1165] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1166] Conventional simulation systems have had difficulty providing simulation results that take into account the user's emotional state. Furthermore, when inputting equipment performance data, there was a lack of means to obtain optimal simulation results based on the user's emotions. This meant that it was not possible to simulate optimal production line speeds or equipment operation under stressful conditions.

[1167] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1168] In this invention, the server includes means for constructing an elaborate virtual environment without requiring programming or skills, means for creating a three-dimensional model from an image, means for making corrections using text instructions, means for inputting equipment performance (load capacity, speed, etc.), means for generating an entire facility in a virtual space, means for simulating the operation of production lines and equipment and the flow of people, means for deriving optimal equipment performance, placement, layout, etc., means for acquiring user emotional data, and means for adjusting the simulation results based on the emotional data. This makes it possible to provide optimal simulation results that take into account the user's emotional state.

[1169] "A means of building an elaborate virtual environment without the need for programming or skills" refers to technology that allows even users without specialized programming knowledge or skills to easily create highly accurate virtual environments.

[1170] "Means for creating a 3D model from an image" refers to technology that automatically generates a 3D model based on 2D image data.

[1171] "Means for modifying with text instructions" refers to a technology that allows a user to modify a virtual environment or a three-dimensional model by inputting instructions in text format.

[1172] "Means for inputting equipment performance (load capacity, speed, etc.)" refers to technology that allows users to input specific performance data of equipment (e.g., load capacity and speed) into the system.

[1173] "Means for generating an entire facility in a virtual space" refers to technology that reproduces an entire facility such as a factory, warehouse, or store in a virtual space.

[1174] "Means for simulating the operation of production lines and equipment, and the flow of people" refers to technology that simulates the operation of production lines, equipment, and the flow of people in a virtual space.

[1175] "Means of deriving optimal equipment performance, placement, layout, etc." refers to technology that determines optimal equipment performance, placement, and layout based on simulation results.

[1176] "Means of acquiring user emotional data" refers to technology that analyzes the user's emotional state from facial expressions, voice, etc., and acquires that data.

[1177] "Means for adjusting simulation results based on emotional data" refers to a technology for adjusting simulation results based on acquired emotional data of a user.

[1178] This invention is a system in which a user inputs performance data of a device, simulates the performance using simulation software, and further adjusts the simulation results based on the user's emotional data. A specific embodiment of this system will be described below.

[1179] First, the user uses a terminal to input the equipment's performance data (e.g., load capacity of 500 kg, maximum speed of 1 m / s) into the system. Input is done through a dedicated interface. This interface is designed to allow users to input data easily.

[1180] The server then receives the performance data sent by the user and launches simulation software (e.g., ANSYS or MATLAB) to simulate the performance of the equipment.

[1181] The server inputs the performance data entered by the user into the launched simulation software. For example, enter data such as "maximum load capacity 500 kg, maximum speed 1 m / s" into ANSYS and run the simulation. While the simulation is running, the server monitors the progress.

[1182] Furthermore, the server obtains the user's current emotional data through the emotion engine, which analyzes the user's stress or relaxation state from their facial expressions and voice. This emotional data is used to reflect the user's emotional state in real time.

[1183] The server adjusts the simulation results based on the acquired emotional data. For example, if the user is feeling stressed, the server generates a simulation result that slows down the production line. This allows the server to provide optimal simulation results that take the user's emotional state into account.

[1184] Finally, the server provides the adjusted simulation results to the user, who can check them through their terminal. As a concrete example, the user inputs the following prompt sentence into the generative AI model:

[1185] Example prompt sentence:

[1186] "The maximum load capacity of equipment A is 500 kg, and its maximum speed is 1 m / s. Please conduct a simulation that takes these performance characteristics into account and show us the results. Also, if the user feels stressed, please provide us with the results of a simulation that reduces the speed of the production line."

[1187] By inputting this prompt sentence, the server uses simulation software to simulate the performance of the device, and uses an emotion engine to provide optimal simulation results according to the user's emotions.

[1188] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1189] Step 1:

[1190] The user inputs the performance data of the device.

[1191] The user opens a dedicated interface on the terminal and inputs "Device A's maximum load capacity is 500 kg, and its maximum speed is 1 m / s." The input data is sent from the terminal to the server. The input data becomes the base data for simulating the device's performance.

[1192] Step 2:

[1193] The server receives the input data and runs the simulation software.

[1194] The server receives the performance data sent by the user. After receiving the data, the server launches the simulation software (e.g., ANSYS or MATLAB). The launched software prepares to run a simulation based on the performance data.

[1195] Step 3:

[1196] The server inputs performance data into the simulation software and executes the simulation.

[1197] The server inputs the performance data entered by the user into the launched simulation software. For example, data such as "maximum load capacity 500 kg, maximum speed 1 m / s" is entered into ANSYS and the simulation is started. As the simulation progresses, the server monitors the progress and obtains the simulation results. As an output, detailed simulation results regarding the operation and performance of the equipment are obtained.

[1198] Step 4:

[1199] The server acquires the user's emotion data from the emotion engine.

[1200] The server obtains the user's current emotional data through the emotion engine. The emotion engine analyzes the user's stress or relaxation state from their facial expressions and voice. The user's facial expressions and voice data are used as input, and the user's emotional state (e.g., stress level) is obtained as output.

[1201] Step 5:

[1202] The server adjusts the simulation results based on the emotion data.

[1203] The server adjusts the simulation results based on the acquired emotional data. For example, if a user is feeling stressed, the server generates a simulation result that slows down the speed of the production line. The emotional data and the simulation results are used as inputs, and the adjusted simulation results are obtained as outputs.

[1204] Step 6:

[1205] The server provides the final simulation results to the user.

[1206] The server generates the adjusted simulation results and sends them to the user's device. The user sees the results on the device screen, which reads, "Device A can withstand a load of 500 kg and can operate at a maximum speed of 1 m / s. However, based on the current stress level, the production line speed will be adjusted to 0.8 m / s." The adjusted simulation results are used as input, and the final simulation results provided to the user are obtained as output.

[1207] (Application example 2)

[1208] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1209] Conventional simulation systems for the operation of production lines and equipment in factories, warehouses, stores, etc. are unable to take into account the emotional state of the user, making it difficult to provide optimal simulation results even in situations where the user is under stress. Furthermore, when feeding back the simulation results to actual business operations, the user's emotional state cannot be reflected, which creates the problem of not reducing user stress in actual operations.

[1210] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1211] In this invention, the server includes means for constructing an elaborate 3D virtual environment without requiring programming or skills, means for creating a 3D model from an image, means for making modifications using text instructions, means for inputting equipment performance (load capacity, speed, etc.), means for generating entire factories, warehouses, stores, etc. in a virtual space, means for simulating the operation of production lines and equipment and the flow of people, means for deriving optimal equipment performance, placement, layout, etc., means for detecting the user's emotional state and adjusting the simulation results, and means for displaying the adjusted simulation results. This makes it possible to provide optimal simulation results that take the user's emotional state into consideration, thereby reducing user stress in actual business operations.

[1212] A "3D virtual environment" is a three-dimensional virtual space generated on a computer, which users can experience visually.

[1213] "Means for creating 3D models from images" refers to a technology that generates three-dimensional solid models based on two-dimensional image data.

[1214] "Means for making modifications based on text instructions" refers to a technology that automatically modifies a 3D model or virtual environment based on text instructions entered by a user.

[1215] "Means for inputting equipment performance" refers to an interface that allows users to input equipment characteristics (load capacity, speed, etc.) into the system.

[1216] "Means of generating in virtual space" refers to technology that reproduces physical spaces such as factories, warehouses, and stores within a virtual environment.

[1217] "Means for simulating the operation of production lines and equipment, and the flow of people" refers to technology that reproduces and simulates the operation of production lines and equipment, and the flow of people, within a virtual environment.

[1218] "Means for deriving optimal equipment performance, placement, layout, etc." refers to technology that proposes optimal equipment performance, placement, and layout based on simulation results.

[1219] The "means for detecting the emotional state of the user and adjusting the simulation results" is a technology for detecting the emotions of the user and adjusting the simulation results based on those emotions.

[1220] The "means for displaying the adjusted simulation results" is a technique for visually displaying to the user the simulation results adjusted based on the emotional state.

[1221] A system for implementing the present invention has the following configuration.

[1222] System Program

[1223] 1. Hardware and Software Use

[1224] Hardware: Smartphone (iOS or Android)

[1225] Software: Python, TensorFlow, Keras, OpenCV, Emotion API (Microsoft Azure, etc.)

[1226] 2. Data Entry

[1227] Users input the factory robot's performance (load capacity, speed, etc.) via a smartphone app.

[1228] Example: "Load capacity: 500kg, Maximum speed: 1m / s"

[1229] 3. Running the simulation

[1230] The server uses Python and TensorFlow to perform simulations based on the input performance data.

[1231] Display simulation results in real time.

[1232] 4. Operation of the Emotion Engine

[1233] Using the smartphone camera, it analyzes the user's face and detects their emotional state using OpenCV and the Emotion API.

[1234] Example: "Users are stressed."

[1235] 5. Adjusting the simulation results

[1236] The server adjusts the simulation results based on the results of the emotion engine.

[1237] Example: "If the user is stressed, the system adjusts the speed of the production line to 0.8 m / s."

[1238] 6. Displaying the results

[1239] The adjusted simulation results are displayed to the user.

[1240] Example: "With the current settings, the robot's speed is 0.8 m / s."

[1241] Program processing explanation

[1242] The server receives factory robot performance data entered by the user through a smartphone app and runs a simulation using Python and TensorFlow. The simulation results are displayed to the user in real time. The server then uses the smartphone's camera to analyze the user's face using OpenCV and the Emotion API to detect their emotional state. Based on this emotional state, the server adjusts the simulation results and displays the adjusted results to the user.

[1243] Examples and prompts

[1244] Specific examples

[1245] Input: "Load capacity: 500kg, Maximum speed: 1m / s"

[1246] Emotional state: "Stress"

[1247] Output: "With the current settings, the robot's speed is 0.8 m / s"

[1248] Prompt Sentence Examples

[1249] The user inputs the performance of the factory robot, for example, "load capacity: 500 kg, maximum speed: 1 m / s." The system then uses the smartphone camera to detect the user's emotional state. For example, if the user is feeling stressed, the system will adjust the production line speed to 0.8 m / s. The adjusted simulation results are then displayed to the user.

[1250] In this way, this system can provide the optimal production environment for users by combining performance simulation of factory robots with emotion adjustment.

[1251] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1252] Step 1:

[1253] The user launches the smartphone app and inputs the factory robot's performance data (load capacity, speed, etc.). The input data is sent from the device to the server. An example input is "load capacity: 500 kg, maximum speed: 1 m / s."

[1254] Step 2:

[1255] The server uses Python and TensorFlow to run a simulation based on the received performance data. Specifically, it receives the performance data as input and simulates the robot's behavior using a generative AI model. The simulation results are generated as output.

[1256] Step 3:

[1257] The server sends the simulation results to the terminal in real time, and the terminal displays the simulation results to the user. An example of the display is "With the current settings, the robot's speed is 1 m / s."

[1258] Step 4:

[1259] The device uses the smartphone camera to capture a picture of the user's face and analyzes the user's emotional state using OpenCV and the Emotion API. The user's face image is used as input, and the emotional state (e.g., stress) is obtained as output.

[1260] Step 5:

[1261] The server adjusts the simulation results based on the results of the emotion engine. Specifically, it receives the emotional state as input and recalculates the simulation results. For example, if the user is feeling stressed, the system adjusts the speed of the production line to 0.8 m / s. The adjusted simulation results are generated as output.

[1262] Step 6:

[1263] The server sends the adjusted simulation results to the terminal, and the terminal displays the adjusted simulation results to the user. An example of the display is "With the current settings, the robot's speed is 0.8 m / s."

[1264] In this way, the system processes and calculates data based on the input data at each step, ultimately providing the user with optimal simulation results.

[1265] Example 3

[1266] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1267] Conventional 3D virtual environment construction systems require programming and specialized skills, making them difficult for general users to use. Furthermore, when simulating production lines, equipment operation, and people flow in virtual space, there is a lack of means to derive optimal equipment performance, placement, and layout. Furthermore, the lack of a function to adjust the virtual environment according to the user's emotions limits the user experience. These issues need to be resolved.

[1268] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1269] In this invention, the server includes a means for constructing an elaborate 3D virtual environment without requiring programming or skills, a means for creating a 3D model from a photograph, a means for making modifications via text instructions, a means for inputting equipment performance (load capacity, speed, etc.), a means for generating an entire facility in a virtual space, a means for simulating the operation of production lines and equipment and the flow of people, a means for deriving optimal equipment performance, placement, layout, etc., and a means for recognizing user emotions and adjusting the 3D virtual environment based on those emotions. This allows even general users to easily construct a 3D virtual environment and derive optimal placement and layout through simulation. Furthermore, adjusting the environment according to the user's emotions can provide a richer user experience.

[1270] A "3D virtual environment" is a three-dimensional virtual space generated by a computer, which can be visually experienced by the user.

[1271] "Method of creating 3D models from photographs" is a technology that generates three-dimensional models based on two-dimensional photographic data.

[1272] "Means for modifying by text instructions" refers to a technique for modifying objects and layouts within a 3D virtual environment based on text instructions entered by a user.

[1273] "Means for inputting equipment performance" refers to a technique for inputting performance data (for example, load capacity and speed) of objects used in a virtual environment.

[1274] "Means for generating entire facilities in virtual space" refers to technology that reproduces physical facilities such as factories, warehouses, and stores in virtual space.

[1275] "Means for simulating the operation of production lines and equipment, and the flow of people" refers to technology that simulates the operation of production lines and equipment, and the flow of people, in a virtual space.

[1276] "Means for deriving optimal equipment performance, placement, layout, etc." refers to technology that proposes optimal equipment performance, placement, and layout based on simulation results.

[1277] "Means for recognizing a user's emotions and adjusting a 3D virtual environment based on those emotions" refers to technology that analyzes a user's emotions and adjusts elements within the 3D virtual environment (such as music or lighting) according to those emotions.

[1278] MODE FOR CARRYING OUT THE INVENTION

[1279] This invention is a system that allows users to build sophisticated 3D virtual environments without the need for programming or specialized skills, and derive optimal placement and layout through simulation. It also provides the ability to adjust the virtual environment according to the user's emotions.

[1280] Virtual space generation and simulation

[1281] The server uses a 3D engine such as Unity or Unreal Engine to generate the entire facility, such as a factory, warehouse, or store, specified by the user, in a virtual space. The user can visualize this virtual space using dedicated terminals and software. For example, the user can input the layout and placement information of the space they want to generate through a GUI on their terminal.

[1282] Users can use simulation software such as AnyLogic or FlexSim to simulate production lines, equipment operation, and people flow. Users set simulation parameters and run the simulation. The server analyzes the simulation results and derives optimal equipment performance, placement, and layout. The analysis results are displayed in the form of dashboards and reports that users can use.

[1283] For example, if a user wants to optimize the layout of a store, they can input the following prompt into the generative AI model:

[1284] "Please suggest the optimal layout based on the results of a store's people flow simulation."

[1285] Coordinating 3D Virtual Environments with Emotion Engines

[1286] The server recognizes the user's emotions using an emotion engine, which is built using a general emotion analysis API, for example. While the user is active in the virtual space, the emotion engine analyzes the user's emotions in real time from their facial expressions and voice.

[1287] The server adjusts the 3D virtual environment based on the recognized emotion. For example, when the user feels happy, the server changes the music in the 3D virtual environment to bright and cheerful. Conversely, when the user feels sad, the server changes the music to quiet and calm.

[1288] For example, a prompt that indicates a user is feeling happy might look like this:

[1289] "Change the music in the 3D virtual environment to something upbeat and fun, as users are feeling the joy."

[1290] In this way, the system provides an optimal virtual environment according to the user's emotions. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1291] Virtual space generation and simulation

[1292] Step 1:

[1293] Users use dedicated terminals and software to request the creation of virtual spaces such as factories, warehouses, and stores. As input, the user enters the layout and placement information of the space they want to create through the GUI on their terminal. As output, this information is sent to the server.

[1294] Step 2:

[1295] The server uses a 3D engine such as Unity or Unreal Engine to generate a virtual space based on information specified by the user. As input, it receives layout information sent by the user and processes the data to create a 3D model. As output, the data of the generated virtual space is sent to the device.

[1296] Step 3:

[1297] The user visually checks the generated virtual space using a dedicated device or software. The input is the virtual space data sent from the server, and the output is the virtual space displayed on the device. Specifically, the user uses a VR headset to move freely within the virtual space and check the layout and placement.

[1298] Step 4:

[1299] Using simulation software such as AnyLogic or FlexSim, users can start simulating the operation of production lines and equipment, as well as the flow of people. Simulation parameters are set as input, and simulation results are generated as output. Specifically, users operate the GUI of the simulation software to input parameters.

[1300] Step 5:

[1301] The server analyzes the simulation results and derives optimal equipment performance, placement, and layout. It receives the simulation results as input and performs analysis as data calculations. The output is an optimized proposal. Specifically, the server executes the analysis algorithm and generates the results in the form of a dashboard or report.

[1302] Step 6:

[1303] The user checks the proposals provided by the server in the form of a dashboard or report. The proposed data sent from the server is received as input, and the proposals are displayed on the terminal as output. Specifically, the user readjusts the layout and placement of the virtual space based on the proposals.

[1304] Coordinating 3D Virtual Environments with Emotion Engines

[1305] Step 1:

[1306] The user begins to act in the virtual space. The input is data from the virtual space, and the output is activity within the virtual space. Specifically, the user puts on a VR headset and explores the virtual space.

[1307] Step 2:

[1308] The server uses an emotion engine to recognize the user's emotions. It receives the user's facial expressions and voice data as input and performs emotion analysis as data processing. The output is the recognized emotion data. Specifically, the server uses an emotion analysis API to analyze emotions in real time.

[1309] Step 3:

[1310] The server adjusts the 3D virtual environment based on the recognized emotions. It receives emotion data as input and performs environmental adjustments as data calculations. The adjusted environmental data is generated as output. Specifically, the server changes the music and lighting in the environment through Unity or Unreal Engine.

[1311] Step 4:

[1312] The user experiences a tailored 3D virtual environment. The tailored environment data is received as input, and the tailored environment is displayed on the device as output. The user continues to work in an optimal environment according to their own emotions.

[1313] (Application example 3)

[1314] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1315] Conventional 3D virtual environment creation systems do not adjust the environment based on the user's emotions, resulting in a poor user experience. Furthermore, simulations that take emotional factors into account were not performed when optimizing store or factory layouts, making it difficult to provide an optimal environment. Furthermore, the system required programming and specialized skills, making it difficult for general users to use.

[1316] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1317] In this invention, the server includes means for constructing an elaborate 3D virtual environment without requiring programming or skills, means for creating a 3D model from an image, means for making modifications using text instructions, means for inputting equipment performance (load capacity, speed, etc.), means for generating an entire factory, warehouse, store, etc. in a virtual space, means for simulating the operation of production lines and equipment and the flow of people, means for deriving optimal equipment performance, placement, layout, etc., means for recognizing a user's emotions and adjusting the virtual environment based on those emotions, and means for capturing the user's emotions using a smart device and adjusting the music and lighting in the virtual environment. This makes it possible to provide an optimal virtual environment according to the user's emotions and optimize the layout of a store or factory by taking emotional factors into consideration.

[1318] "A means of creating sophisticated 3D virtual environments without the need for programming or skills" is a technology that allows users to easily create high-quality 3D virtual spaces without having specialized knowledge or skills.

[1319] "Means for creating 3D models from images" refers to a technology that automatically generates three-dimensional 3D models based on photographs or image data.

[1320] "Means for modifying by text instructions" refers to a technique that allows a user to modify objects and layouts in a 3D virtual environment by inputting instructions in text format.

[1321] "Means for inputting equipment performance (load capacity, speed, etc.)" refers to a technique for inputting performance data of objects and equipment used within a virtual environment.

[1322] "Means for generating entire factories, warehouses, stores, etc. in virtual space" refers to technology that reproduces actual physical spaces such as factories, warehouses, and stores in virtual space.

[1323] "Means for simulating the operation of production lines and equipment, and the flow of people" refers to technology that simulates the operation of production lines and equipment, and the flow of people, in a virtual space.

[1324] "Means for deriving optimal equipment performance, placement, layout, etc." refers to technology that proposes optimal equipment performance, placement, and layout based on simulation results.

[1325] "Means for recognizing a user's emotions and adjusting the virtual environment based on those emotions" refers to technology that detects a user's emotions and adjusts elements in the virtual environment (such as music or lighting) according to those emotions.

[1326] "Means for capturing a user's emotions using a smart device and adjusting music and lighting in a virtual environment" refers to technology that uses a device such as a smartphone or tablet to capture a user's emotions and change the music and lighting in a virtual environment based on those emotions.

[1327] A system for carrying out this invention has the following configuration: The server includes means for constructing an elaborate 3D virtual environment without the need for programming or skills, means for creating a 3D model from an image, means for making modifications using text instructions, means for inputting equipment performance (load capacity, speed, etc.), means for generating entire factories, warehouses, stores, etc. in a virtual space, means for simulating the operation of production lines and equipment and the flow of people, means for deriving optimal equipment performance, placement, layout, etc., means for recognizing user emotions and adjusting the virtual environment based on those emotions, and means for capturing user emotions using a smart device and adjusting the music and lighting in the virtual environment.

[1328] Hardware and software used

[1329] Hardware: smartphones, tablets, cameras, speakers

[1330] Software: OpenCV (image capture), Emotion Recognition API (emotion recognition), Unity (3D virtual environment generation and adjustment)

[1331] Data processing and calculation

[1332] 1. Image capture: When a user is using a smartphone or tablet, the camera captures an image of the user's face.

[1333] 2. Emotion Recognition: The captured image is sent to the emotion recognition API to recognize the user's emotions. The emotion recognition API analyzes the image data and identifies the user's emotions (e.g., joy, sadness, etc.).

[1334] 3. Virtual environment adjustment: Adjust the music and lighting in the Unity environment based on the recognized emotion, for example, change the music and lighting to brighter if the user is happy, or change the music and lighting to quieter if the user is sad.

[1335] Specific examples

[1336] When a user is using their smartphone in a store, the camera captures their face and recognizes their emotions. If the user is happy, the music in the store will change to a bright and cheerful one and the lights will be brighter. Conversely, if the user is sad, the music will change to a quiet and calm one and the lights will be dim.

[1337] Prompt Sentence Examples

[1338] When a user uses a smartphone, capture their face with the camera and recognize their emotions using the emotion recognition API. Based on the recognized emotion, adjust the music and lighting in the Unity environment. For example, if the user is happy, use bright music and lighting, and if the user is sad, use quiet music and subdued lighting.

[1339] In this way, it becomes possible to provide an optimal virtual environment according to the user's emotions and to optimize the layout of stores and factories by taking emotional factors into account.

[1340] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1341] Step 1:

[1342] When a user uses a smartphone, the device's camera captures an image of the user's face.

[1343] Input: Smartphone camera image

[1344] Output: Captured user's face image

[1345] Specific behavior: The smartphone's camera app launches, detects the user's face, and captures an image.

[1346] Step 2:

[1347] The device sends the captured facial image to an emotion recognition API to recognize the user's emotions.

[1348] Input: A captured image of the user's face

[1349] Output: Emotion data returned from the emotion recognition API (e.g., joy, sadness, etc.)

[1350] Specific operation: The device sends the captured image data to the emotion recognition API and receives the emotion data returned by the API.

[1351] Step 3:

[1352] The server adjusts the music and lighting in the virtual environment based on the emotional data received from the emotion recognition API.

[1353] Input: Emotion data from the emotion recognition API

[1354] Output: Adjusted virtual environment settings (music, lighting, etc.)

[1355] Specific operation: The server analyzes the emotional data and changes the settings to bright music and lighting if the user is happy, and to quiet music and subdued lighting if the user is sad.

[1356] Step 4:

[1357] The server transmits the adjusted settings of the virtual environment to the terminal, and the terminal displays the virtual environment based on the settings.

[1358] Input: Adjusted virtual environment settings

[1359] Output: The adjusted virtual environment displayed on the user's smartphone.

[1360] Specific operation: The server sends the adjusted configuration data to the terminal, and the terminal receives the data and updates the virtual environment.

[1361] Step 5:

[1362] The user experiences the adjusted virtual environment and provides feedback as needed.

[1363] Input: Adjusted virtual environment

[1364] Output: User feedback (e.g. satisfied, dissatisfied, etc.)

[1365] How it works: Users experience the calibrated virtual environment on their smartphone and provide feedback and suggestions for improvement using the in-app feedback feature.

[1366] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1367] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1368] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1369] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1370] [Third embodiment]

[1371] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1372] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1373] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1374] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1375] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1376] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1377] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1378] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1379] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1380] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1381] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1382] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1383] "Example 1"

[1384] In one embodiment of the present invention, a user first inputs a photograph into the system. This photograph may be, for example, a photograph of the interior or exterior of a factory, a product, or equipment. The system then automatically generates a 3D model from the photograph. The generated 3D model can be modified based on the user's text instructions. For example, the 3D model can be easily modified by inputting instructions into the system, such as "move equipment A two meters to the right" or "change the wall color to blue."

[1385] "Example 2"

[1386] Next, the user inputs the equipment's performance (load capacity, speed, etc.) into the system. This allows the system to accurately simulate the operation and performance of the equipment. For example, by inputting information such as the maximum load capacity of equipment A being 500 kg and the maximum speed being 1 m / s into the system, it is possible to perform a simulation that takes these performances into account. "Example 3"

[1387] Furthermore, users can generate entire factories, warehouses, stores, etc. in virtual space, allowing them to visually grasp the overall layout and arrangement. In addition, they can also visualize the operation of production lines and equipment.

[1388] By simulating people flow, it is possible to derive optimal equipment performance, placement, layout, etc. For example, the system can automatically generate proposals for optimizing store layouts based on the results of people flow simulations.

[1389] The processing flow of each embodiment will be described below.

[1390] "Example 1"

[1391] Step 1: The user enters photos of the factory interior, exterior, products, equipment, etc. into the system.

[1392] Step 2: The system automatically generates a 3D model from the input photo.

[1393] Step 3: The user modifies the generated 3D model based on text instructions, such as "move device A 2 meters to the right" or "color the wall blue."

[1394] "Example 2"

[1395] Step 1: The user inputs the equipment's performance (load capacity, speed, etc.) into the system.

[1396] Step 2: The system simulates the operation and performance of the equipment based on the input performance information. For example, input information such as the maximum load capacity of Equipment A being 500 kg and the maximum speed being 1 m / s into the system.

[1397] "Example 3"

[1398] Step 1: The user generates an entire factory, warehouse, store, etc. in a virtual space.

[1399] Step 2: The system simulates the operation of production lines, equipment, and people flow in the generated virtual space.

[1400] Step 3: Based on the simulation results, the system derives optimal equipment performance, placement, layout, etc. For example, it automatically generates proposals for optimizing store layouts based on the results of a people flow simulation.

[1401] Example 1

[1402] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1403] Conventional 3D model generation systems require specialized programming and skills, making them difficult for general users to use. Furthermore, there were limitations on the accuracy of creating 3D models from photographs and the flexibility of modification based on user instructions. Furthermore, there was a lack of a way to feed back simulation results based on the generated 3D models into actual business operations.

[1404] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1405] In this invention, the server includes: means for inputting images captured by a user; means for the server to analyze the input images; means for the server to generate a three-dimensional model based on the analyzed data; means for the user to input text instructions; means for the server to analyze the text instructions and modify the three-dimensional model; means for inputting equipment performance; means for generating an entire factory, warehouse, store, etc. in a virtual space; means for simulating the operation of production lines and equipment and the flow of people; and means for deriving optimal equipment performance, placement, layout, etc. This allows users without specialized skills to easily generate and flexibly modify highly accurate three-dimensional models. Furthermore, the simulation results can be fed back into actual business operations.

[1406] A "user" is an individual or corporation that uses the system to input images and generate or modify three-dimensional models.

[1407] A "server" is a computer system that analyzes images input by a user and generates and modifies three-dimensional models.

[1408] "Images" refer to photographs or images taken by users and entered into the system.

[1409] "Analysis" means that the server extracts feature points, object contours, colors, etc. from the input image and processes them as data.

[1410] A "three-dimensional model" is a three-dimensional digital representation generated based on analyzed data.

[1411] A "text instruction" is a sentence that describes the details of the corrections or changes that the user will make to the three-dimensional model.

[1412] "Natural language processing" is a technology that allows a server to understand text instructions from a user and make appropriate corrections.

[1413] A "virtual space" is a simulated environment that digitally recreates the entirety of a factory, warehouse, store, etc.

[1414] "Simulation" is a technology that reproduces production lines, equipment operations, people flow, etc. in a virtual space to derive optimal placement and layout.

[1415] "Devices" are digital data such as three-dimensional models and simulation results generated by the system.

[1416] "Performance" refers to the physical characteristics of the equipment, such as load capacity and speed.

[1417] The present invention is a system that generates a three-dimensional model based on an image taken by a user and then modifies the model based on text instructions from the user. A specific embodiment of this system will be described below.

[1418] First, the user uses a terminal to upload images of the interior and exterior of the factory, products, equipment, etc. to the system. These images are in common image formats such as JPEG and PNG. The user opens the terminal's browser, accesses the system's upload page, clicks the "Select File" button to select the image to upload, and clicks the "Upload" button to send the image to the system.

[1419] Next, the server receives the uploaded image and analyzes it using image processing software (e.g., OpenCV or TensorFlow). The server receives the image data and launches the image processing software to analyze the image's feature points, object contours, color, etc.

[1420] Based on the analyzed data, the server automatically generates a 3D model using 3D model generation software (e.g., Blender or Autodesk Maya). The server inputs the analysis results into the 3D model generation software, which then constructs a 3D model based on the image information. The generated 3D model is stored on the server and is ready to be displayed to the user.

[1421] The user checks the generated 3D model in the browser on their device and enters specific instructions for the parts they want to modify in the text boxes. For example, they might say, "Move device A two meters to the right" or "Change the wall color to blue." When the user clicks the "Submit" button, the instructions are sent to the system.

[1422] The server analyzes the text instructions from the user using natural language processing software (e.g., GPT-4 or BERT) to understand the content of the instructions. The server receives the text instructions, launches the natural language processing software, analyzes the text instructions, and extracts specific corrections.

[1423] Based on the understood instructions, the server again uses the 3D model generation software to modify the 3D model. The server inputs the analysis results into the 3D model generation software, which modifies the 3D model based on the instructions. The modified 3D model is stored on the server and displayed to the user.

[1424] For example, if a user uploads an image of the inside of a factory to the system, the server receives the image, analyzes it using OpenCV, and generates a 3D model using Blender. Next, if the user inputs the instruction "move equipment A 2 meters to the right," the server analyzes this instruction using GPT-4 and again uses Blender to move equipment A in the 3D model 2 meters to the right.

[1425] An example of a prompt is as follows:

[1426] "Upload a photo of the interior of your factory. Then, if you want to make any modifications to the generated 3D model, enter specific instructions in text. For example, 'Move equipment A two meters to the right,' or 'Change the color of the walls to blue.'"

[1427] The above is a specific embodiment for carrying out the present invention.

[1428] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1429] Step 1:

[1430] A user uploads an image using a device.

[1431] Specific operation: The user opens the browser on the device, accesses the system's upload page, clicks the "Select file" button, selects the image to upload, and clicks the "Upload" button.

[1432] Input: User-taken image (JPEG or PNG format)

[1433] Output: Image data sent to the server

[1434] Step 2:

[1435] The server receives the uploaded images and analyzes them using image processing software.

[1436] How it works: The server receives the image data and launches image processing software such as OpenCV or TensorFlow, which analyzes the image's features, object contours, color, etc.

[1437] Input: Uploaded image data

[1438] Output: Analyzed image data (information such as feature points, contours, and colors)

[1439] Step 3:

[1440] The server generates a three-dimensional model based on the analyzed data using three-dimensional model generation software.

[1441] How it works: The server inputs the analysis results into 3D model generation software such as Blender or Autodesk Maya, which then creates a 3D model based on the image information.

[1442] Input: Analyzed image data

[1443] Output: Generated 3D model

[1444] Step 4:

[1445] The user uses a terminal to check the generated three-dimensional model and inputs correction instructions in text.

[1446] Specific operation: The user checks the 3D model in the browser on their device, enters specific instructions about the parts they want to correct in the text box, and clicks the "Submit" button.

[1447] Input: Generated 3D model, user text instructions

[1448] Output: Text instructions sent to the server

[1449] Step 5:

[1450] The server analyzes the text instructions from the user using natural language processing software.

[1451] Specific operation: The server receives the text instructions and launches natural language processing software such as GPT-4 or BERT. The software analyzes the text instructions and extracts specific corrections.

[1452] Input: User's text instructions

[1453] Output: Parsed modifications

[1454] Step 6:

[1455] The three-dimensional model generation software is then used again to modify the three-dimensional model based on the instructions understood by the server.

[1456] Specific operation: The server inputs the analysis results into the 3D model generation software. The software modifies the 3D model based on the instructions. The server saves the modified 3D model and displays it to the user.

[1457] Input: Analyzed corrections, generated 3D model

[1458] Output: Modified 3D model

[1459] The above is the specific flow of the program processing of this system.

[1460] (Application example 1)

[1461] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1462] Conventionally, optimizing the layout and equipment placement of factories, warehouses, stores, etc. required specialized knowledge and skills, and was problematic in that it was time-consuming and costly. Furthermore, there was no system that could generate a 3D model from on-site photos and modify it based on user instructions. This made it difficult to create a virtual environment that accurately reflected on-site conditions and perform efficient simulations.

[1463] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1464] In this invention, the server includes means for constructing an elaborate 3D virtual environment without requiring programming or skills, means for creating a 3D model from a photograph, means for making modifications based on text instructions, means for inputting equipment performance (load capacity, speed, etc.), means for generating entire factories, warehouses, stores, etc. in a virtual space, means for simulating the operation of production lines and equipment and the flow of people, means for deriving optimal equipment performance, placement, layout, etc., means for taking photographs of the inside of a factory using a smartphone and generating a 3D model from the photograph, means for modifying the generated 3D model based on the user's text instructions, and means for displaying the modified 3D model. This allows users to generate 3D models based on photos of the site and easily modify them, enabling efficient optimization of layout and equipment placement.

[1465] "A means of creating an elaborate 3D virtual environment without the need for programming or skills" is a method that allows even users without specialized knowledge or skills to easily create a highly accurate 3D virtual space.

[1466] "Means for creating 3D models from photographs" refers to a technology that automatically generates 3D models based on photographs taken by users.

[1467] The "means for modifying by text instructions" is a method for modifying a 3D model based on text instructions entered by a user.

[1468] "Means for inputting equipment performance (load capacity, speed, etc.)" refers to a method for inputting performance and characteristics into the generated 3D model.

[1469] "Means for generating entire factories, warehouses, stores, etc. in virtual space" refers to technology that reproduces actual spaces such as factories, warehouses, and stores in virtual space.

[1470] "Means for simulating the operation of production lines and equipment, and the flow of people" refers to a method for simulating the operation of production lines and equipment, and the flow of people in a virtual space.

[1471] "Means for deriving optimal equipment performance, placement, layout, etc." refers to a method for determining optimal equipment performance, placement, and layout based on simulation results.

[1472] "A means of taking photos of the inside of a factory using a smartphone and generating a 3D model from those photos" is a technology that generates a 3D model based on photos of the inside of a factory taken with a smartphone.

[1473] The "means for modifying the generated 3D model based on the user's text instructions" is a method for modifying the generated 3D model according to the text instructions entered by the user.

[1474] The "means for displaying the modified 3D model" is a technology for visually displaying the modified 3D model.

[1475] A system for implementing this invention is configured as follows: First, a user takes a photo of the inside of a factory using a smartphone. The taken photo is input into the system, which automatically generates a 3D model from the photo. The generated 3D model can be modified based on the user's text instructions. For example, possible instructions include "move machine A 3 meters to the left" or "change the color of the wall to green." The modified 3D model is visually displayed, allowing the user to confirm the results.

[1476] This system is realized using the following hardware and software: a smartphone, a server, and a display device. The software uses OpenCV, Blender, and OpenAI's GPT-3.

[1477] The specific process is as follows: First, a photo taken with a smartphone is sent to the server. The server analyzes the photo using OpenCV and generates a 3D model. The generated 3D model is visually displayed using Blender. When the user enters text instructions, the instructions are sent to the server and analyzed using OpenAI's GPT-3. Based on the analysis results, the 3D model is modified and displayed again in Blender.

[1478] As a concrete example, consider a scenario in which the layout of machines in a factory needs to be changed. A user takes a photo of the factory and inputs the instruction "move machine A 3 meters to the left." The system generates a 3D model from the photo and repositions machine A based on the instruction. The modified 3D model is visually displayed to the user, allowing the user to confirm the results.

[1479] An example of a prompt is as follows:

[1480] "Take a photo of your factory and enter it into the app. Then enter instructions like this:

[1481] "Move machine A 3 meters to the left"

[1482] Change the wall color to green

[1483] "Move product B forward 2 meters."

[1484] In this way, users can easily optimize the layout of their factory.

[1485] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1486] Step 1:

[1487] A user takes a photo of the inside of a factory using a smartphone. The photo is saved in the smartphone. The input is the photo of the inside of the factory, and the output is the image file saved in the smartphone.

[1488] Step 2:

[1489] A user starts the application on their smartphone and uploads a photograph they have taken to the system. The input is the image file on the smartphone, and the output is the image data sent to the server.

[1490] Step 3:

[1491] The server analyzes the received image data using OpenCV and generates a 3D model from the photo. The input is the image data sent to the server, and the output is the generated 3D model data. Specifically, it extracts feature points from the image and reconstructs the 3D shape based on them.

[1492] Step 4:

[1493] The server visually displays the generated 3D model using Blender. The input is the generated 3D model data, and the output is the visually displayed 3D model. Specifically, it uses Blender's API to render the 3D model.

[1494] Step 5:

[1495] The user inputs text instructions through the application, for example, "move machine A 3 meters to the left." The input is the user's text instructions, and the output is the text data sent to the server.

[1496] Step 6:

[1497] The server uses OpenAI's GPT-3 to analyze the received text instructions. The input is the text data sent to the server, and the output is the analysis result. Specifically, the text instructions are processed using the GPT-3 API for natural language processing.

[1498] Step 7:

[1499] The server modifies the 3D model based on the analysis results. The input is the analysis results and the generated 3D model data, and the output is the modified 3D model data. Specifically, it uses Blender's API to change the position and color of the 3D model.

[1500] Step 8:

[1501] The server visually displays the modified 3D model again using Blender. The input is the modified 3D model data, and the output is the visually displayed modified 3D model. Specifically, the server renders the modified 3D model using Blender's API.

[1502] Step 9:

[1503] The user checks the corrected 3D model through the application. The input is the visually displayed corrected 3D model, and the output is the user's confirmation result. Specifically, the corrected 3D model is displayed on the smartphone display.

[1504] Example 2

[1505] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1506] In conventional systems, advanced programming skills were required to build an elaborate virtual environment, and complex settings were often required to run simulations based on equipment performance data. This made it difficult for users to easily build a virtual environment and simulate equipment performance. Furthermore, there was a lack of a way to feed back the simulation results into actual business operations.

[1507] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1508] In this invention, the server includes means for receiving information on the performance of equipment moving between factories, warehouses, and stores, means for generating a virtual space representing the factories, warehouses, and stores, a prompt statement for instructing a simulation of the movement of the equipment in the virtual space based on the performance data of the equipment, means for executing a simulation of the movement of the equipment using a generative AI model, and means for providing a user with the simulation results. This enables a user without advanced skills to build a virtual environment, simulate the performance of equipment, and reflect the results in actual business operations. The generating means in the server generates a three-dimensional model representing the factory, warehouse, and store in a virtual space from photographs representing the factory, warehouse, and store. The executing means in the server executes a simulation of the movement of the equipment using the prompt sentence, the generating AI model, and the simulation software.

[1509] A "virtual environment" is a digital space generated on a computer that mimics the real world.

[1510] A "three-dimensional model" is digital data that represents the shape and structure of an object in three-dimensional space.

[1511] A "text instruction" is an instruction for an operation or correction given to the system by the user using text information.

[1512] "Equipment performance data" refers to data that indicates the physical characteristics, such as the load capacity and speed, of equipment moving between factories, warehouses, and stores.

[1513] A "generative AI model" is a model that uses artificial intelligence technology to set up simulations based on equipment performance data.

[1514] A "prompt" is a text sentence that instructs the generative AI model on the simulation settings and conditions.

[1515] "Simulation" is the process of simulating the operation and performance of equipment in a virtual environment.

[1516] "Feedback" is the process of reflecting the results of the simulation in actual business operations.

[1517] This invention is a system in which a user inputs performance data of a device, a server uses a generated AI model to perform a simulation based on that data, and provides the results to the user. A specific embodiment of this system will be described below.

[1518] Hardware and software used

[1519] Hardware: Servers, user terminals

[1520] Software: Generative AI models, simulation software

[1521] System processing flow

[1522] First, three-dimensional models representing the factories, warehouses, and stores are generated in a virtual space from photographs of the factories, warehouses, and stores in the same manner as described in the first embodiment. Then, the following processing is performed. 1. The user inputs the device performance data.

[1523] The user inputs the performance data of the device into the input form on the terminal. For example, the maximum load capacity of device A is 500 kg, and the maximum speed is 1 m / s.

[1524] 2. The device sends the entered performance data to the server.

[1525] The terminal converts the performance data entered by the user into a data format such as JSON and sends an HTTP request to the server.

[1526] 3. The server receives the performance data and generates a prompt for the generative AI model.

[1527] The server analyzes the received performance data and generates a prompt to input to the generative AI model. For example, it generates a prompt such as, "Please perform an operation simulation when the maximum load capacity of device A is 500 kg and the maximum speed is 1 m / s."

[1528] 4. The server uses the generated AI model to set up the simulation.

[1529] The server inputs a prompt statement into the generative AI model to generate the settings necessary for the simulation. The generative AI model then sets the simulation parameters and conditions based on the prompt statement.

[1530] 5. The server runs the simulation using the simulation software.

[1531] The server launches the simulation software and runs a simulation based on the settings obtained from the generative AI model. The simulation software calculates the behavior and performance of the equipment and generates simulation results.

[1532] 6. The server sends the simulation results to the device.

[1533] The server converts the simulation results into a data format such as JSON and sends them to the terminal as an HTTP response.

[1534] 7. The device displays the simulation results to the user.

[1535] The device analyzes the received simulation results and displays them in a format that is easy for the user to view. For example, the simulation results may show, in graphs and text, how device A behaves when it supports a load of 500 kg or when it operates at a speed of 1 m / s.

[1536] Specific examples

[1537] Prompt Sentence Examples

[1538] "Please perform a simulation of the operation of device A when the maximum load capacity is 500 kg and the maximum speed is 1 m / s."

[1539] Simulation result display example

[1540] "Device A can support a load of 500 kg and has been confirmed to operate stably at a speed of 1 m / s."

[1541] In this way, the server uses the generated AI model and simulation software to simulate the operation and performance of equipment based on performance data entered by the user, and provides the results to the user.This system enables users to create a virtual environment, simulate equipment performance, and reflect the results in actual business operations without requiring advanced skills.

[1542] The flow of the identification process in the second embodiment will be described with reference to FIG. First, a three-dimensional model representing the factory, warehouse, and store is generated in a virtual space from photographs of the factory, warehouse, and store. Then, the following steps are performed.

[1543] Step 1:

[1544] The user inputs the performance data of the device.

[1545] Input: The user enters the equipment's performance data (e.g., maximum load capacity of 500 kg, maximum speed of 1 m / s) into the terminal's input form.

[1546] Specific operation: The user uses the keyboard to enter device performance data into a form displayed on the terminal screen.

[1547] Step 2:

[1548] The terminal transmits the input performance data to the server.

[1549] Input: User-entered performance data.

[1550] Output: JSON formatted data sent to the server.

[1551] Specific operation: The terminal converts the performance data entered by the user into JSON format, creates an HTTP request, and sends it to the server.

[1552] Step 3:

[1553] The server receives the performance data and generates prompt sentences for the generative AI model.

[1554] Input: Performance data sent from the device in JSON format.

[1555] Output: A prompt to be fed into the generative AI model.

[1556] Specific operation: The server analyzes the received performance data and generates a prompt message saying, "Please perform an operation simulation when the maximum load capacity of device A is 500 kg and the maximum speed is 1 m / s."

[1557] Step 4:

[1558] The server uses the generated AI model to set up the simulation.

[1559] Input: The generated prompt statement.

[1560] Output: The configuration parameters required for the simulation.

[1561] Specific operation: The server inputs a prompt statement into the generative AI model, and the generative AI model sets the simulation parameters and conditions based on the prompt statement.

[1562] Step 5:

[1563] The server runs the simulation using the simulation software.

[1564] Input: Simulation setup parameters obtained from the generative AI model.

[1565] Output: Simulation results.

[1566] Specific operation: The server starts the simulation software, inputs the setting parameters, and runs the simulation. The simulation software calculates the behavior and performance of the equipment and generates the simulation results.

[1567] Step 6:

[1568] The server transmits the simulation results to the terminal.

[1569] Input: Simulation results.

[1570] Output: Simulation result data in JSON format sent to the terminal.

[1571] Specific operation: The server converts the simulation results into JSON format and sends them to the terminal as an HTTP response.

[1572] Step 7:

[1573] The terminal displays the simulation results to the user.

[1574] Input: Simulation result data in JSON format sent from the server.

[1575] Output: Simulation results that are displayed to the user.

[1576] Specific operation: The device analyzes the received simulation results and displays them in a format that is easy for the user to view. For example, the simulation results may show the behavior of device A when it supports a load of 500 kg or when it operates at a speed of 1 m / s, as graphs or text.

[1577] (Application example 2)

[1578] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1579] With conventional systems, building virtual environments for factories, warehouses, stores, etc. required programming and specialized skills, making them difficult for general users to use. Furthermore, there was a lack of means to accurately simulate equipment performance, making it difficult to derive optimal equipment performance and placement for actual operation. Furthermore, there was no way to visually check the simulation results, making it difficult for users to intuitively understand the results.

[1580] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1581] In this invention, the server includes means for constructing an elaborate 3D virtual environment without requiring programming or skills, means for creating a 3D model from an image, means for making modifications using text instructions, means for inputting equipment performance (load capacity, speed, etc.), means for generating entire factories, warehouses, stores, etc. in a virtual space, means for simulating the operation and flow of production lines and equipment, means for deriving optimal equipment performance, placement, layout, etc., and means for simulating and visually displaying the operation and performance of robots based on performance data input by the user. This enables users to easily construct a virtual environment without specialized skills, accurately simulate equipment performance, and visually confirm it.

[1582] A "3D virtual environment" is a three-dimensional virtual space generated on a computer, which users can experience visually.

[1583] "Means for creating 3D models from images" refers to a technology that generates three-dimensional solid models based on two-dimensional image data.

[1584] "Means for making corrections based on text instructions" is a technology that automatically makes corrections to the virtual environment or model based on text information entered by the user.

[1585] The "means for inputting equipment performance" is an interface that allows the user to input equipment characteristics (such as load capacity and speed) into the system.

[1586] "Means of generating in virtual space" refers to technology that reproduces real-world environments such as factories, warehouses, and stores in virtual space.

[1587] "Means of simulation" refers to technology that calculates behavior and performance within a virtual environment and mimics actual behavior.

[1588] "Means for deriving optimal equipment performance, placement, layout, etc." refers to technology that calculates the most efficient equipment performance, placement, and layout based on simulation results.

[1589] "Visual display means" refers to technology that graphically displays simulation results, allowing users to intuitively understand them.

[1590] This invention is a system that allows users without specialized skills to easily create a 3D virtual environment, accurately simulate the performance of equipment, and visually check it. A specific embodiment of this system is described below.

[1591] System configuration

[1592] The system consists of the following main components:

[1593] 1. Server: Manages the entire system and processes data.

[1594] 2. User terminal: Provides an interface for user operation.

[1595] 3. Software: Programs for building, simulating and visualizing 3D virtual environments.

[1596] Hardware and software used

[1597] Hardware: Regular PC or tablet

[1598] Software: Python, matplotlib, numpy

[1599] Program processing explanation

[1600] The server receives performance data (such as load capacity and speed) entered by the user and performs a simulation based on this data. Specifically, the process proceeds as follows:

[1601] 1. User input: The user inputs the device's performance data through a terminal, such as "maximum load capacity 500 kg, maximum speed 1 m / s."

[1602] 2. Data processing: The server performs a simulation based on the received performance data. It uses the Python numpy library to calculate the change in speed and load over time.

[1603] 3. Visualization: The simulation results are displayed graphically using the matplotlib library, which allows users to intuitively understand the results.

[1604] Specific examples

[1605] For example, if a user wants to simulate a robot with a maximum load capacity of 500 kg and a maximum speed of 1 m / s, they would enter the following:

[1606] python

[1607] max_load = 500 kg

[1608] max_speed = 1 m / s

[1609] The server runs a simulation based on this data and visually displays the results, allowing users to intuitively understand the robot's behavior and performance.

[1610] Prompt Sentence Examples

[1611] "Please simulate the movement of a robot with a maximum load capacity of 500 kg and a maximum speed of 1 m / s."

[1612] In this way, users can easily create a virtual environment, simulate the performance of equipment, and visually check it, even if they do not have specialized skills.

[1613] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1614] Step 1:

[1615] The user uses a terminal to input the performance data of the equipment (for example, maximum load capacity and maximum speed). The input data is sent to the server. An example of input data is "maximum load capacity 500 kg, maximum speed 1 m / s."

[1616] Step 2:

[1617] The server prepares the simulation based on the received performance data. Specifically, it uses the Python numpy library to generate the time axis data required for the simulation. Based on the input data, time axis data (for example, 100 steps from 0 seconds to 10 seconds) is generated.

[1618] Step 3:

[1619] The server calculates the change in velocity and load based on the generated time axis data. Specifically, it uses the numpy library to calculate the change in velocity and load over time. For example, the velocity is calculated to increase linearly with time, and the load is calculated to increase exponentially with time.

[1620] Step 4:

[1621] The server generates simulation results based on the calculation results. Specifically, it uses the calculated velocity and load data and stores the simulation results in a data structure. The simulation results include velocity and load values ​​at each time step.

[1622] Step 5:

[1623] The server generates graphs using the matplotlib library to visualize the simulation results. Specifically, it plots the changes in speed and load against time and creates a graph to visually display them. The generated graph visually shows the changes in speed and load.

[1624] Step 6:

[1625] The server sends the generated graphs to the user's device, where the user can visually check the simulation results, allowing them to intuitively understand the robot's behavior and performance.

[1626] Step 7:

[1627] Based on the visualized simulation results, users can re-enter performance data as needed and run the simulation again, enabling iterative simulations to derive optimal equipment performance and placement.

[1628] Example 3

[1629] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1630] Determining efficient layouts and equipment placements is extremely important in modern physical spaces such as factories, warehouses, and stores. However, achieving this requires advanced specialized knowledge and skills, and repeated trial and error can be time-consuming and costly. Furthermore, there is a lack of means to analyze simulation results and derive optimal placements and layouts, making efficient operation difficult.

[1631] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1632] In this invention, the server includes means for constructing an elaborate 3D virtual environment without requiring programming or skills, means for creating a 3D model from an image, means for making modifications via text instructions, means for inputting equipment performance (load capacity, speed, etc.), means for generating entire factories, warehouses, stores, etc. in a virtual space, means for simulating the operation of production lines and equipment and the flow of people, means for deriving optimal equipment performance, placement, layout, etc., means for using a generative AI model to analyze the simulation results, and means for automatically generating optimization proposals based on the analysis results. This enables users to determine efficient layouts and equipment placements without requiring advanced specialized knowledge or skills.

[1633] A "3D virtual environment" is a three-dimensional virtual space generated on a computer, in which users can visually recreate and manipulate physical space.

[1634] "Means for creating 3D models from images" refers to a technology that generates three-dimensional models based on two-dimensional image data, and is a means that can create three-dimensional objects from photographs or drawings.

[1635] "Means for modifying by text instructions" refers to a technique that allows a user to modify objects and layouts in a virtual environment by inputting instructions in text format.

[1636] "Means for inputting equipment performance" refers to a technique for inputting performance data (for example, load capacity, speed, etc.) of objects and equipment used in a virtual environment.

[1637] "A means of generating entire factories, warehouses, stores, etc. in virtual space" is a technology that reproduces the entire physical space in a three-dimensional virtual environment, allowing users to visually grasp that space.

[1638] "Means for simulating the operation of production lines and equipment, and the flow of people" refers to technology that simulates the operation of production lines, the movement of equipment, and the flow of people within a virtual environment, and is a means that can simulate actual operations.

[1639] "Means for deriving optimal equipment performance, placement, layout, etc." refers to technology that automatically proposes optimal equipment performance, placement, and layout based on simulation results.

[1640] "Means using generative AI models" refers to technology that uses artificial intelligence models to analyze data and generate results, such as means for analyzing simulation results and proposing optimization.

[1641] "Means for automatically generating optimization proposals based on analysis results" refers to technology that automatically generates proposals for optimal placement and layout based on the analysis results obtained by a generative AI model.

[1642] This invention provides a system that allows a user to generate a physical space such as a factory, warehouse, or store in a virtual space and determine an efficient layout and arrangement of equipment. Specific embodiments of this system are described below.

[1643] Hardware and software used

[1644] Hardware: PC with a high-performance graphics card, VR headset

[1645] Software: Unity (game engine), Python (programming language), generative AI models (e.g., OpenAI's GPT-4)

[1646] Virtual space generation

[1647] Users use Unity to recreate physical spaces such as factories, warehouses, and stores in a virtual space. Specifically, they use Unity's editor to design the building structure and internal layout. For example, they can set the placement of walls, floors, and equipment using drag and drop.

[1648] Entering data

[1649] Users use Python scripts to input production line and equipment operation and people flow data. Specifically, people flow data stored in a CSV file is read using Python and passed to Unity. For example, the pandas library is used to read the CSV file and format the data.

[1650] Running the simulation

[1651] The server runs the simulation on Unity. Specifically, it launches Unity's simulation engine and visualizes the operation of the prod...

Claims

1. means for receiving information regarding the performance of equipment moving between factories, warehouses, and stores; A means for generating in a virtual space a factory, a warehouse, and a store; a means for executing a simulation of the movement of the device by using a prompt sentence for instructing to simulate the movement of the device in the virtual space based on performance data of the device and a generative AI model; A means for providing the simulation results to a user; A system including:

2. 2. The system according to claim 1, wherein said generating means generates a three-dimensional model representing the factory, warehouse, and store in a virtual space from photographs representing the factory, warehouse, and store.

3. The system according to claim 1 , wherein the executing means executes a simulation of the movement of the equipment using the prompt sentence, the generative AI model, and the simulation software.

Images