System

Abstract

A system is provided.SOLUTION: A system comprising: a depth sensor means for scanning an object; a processing means for processing data obtained from the depth sensor means to generate a three dimensional model; an input means for inputting a user request for the generated three dimensional model; a generating means for analyzing the request obtained from the input means and generating a suitable data file based on the three dimensional model; and an output means for transmitting the generated data file to an output device to generate the object.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] This invention addresses the challenge of creating something that perfectly fits a specific place or object in everyday life. Specifically, there is a need to provide an environment where users can easily customize and create items of sizes or shapes that do not fit commercially available products. In particular, existing technologies require specialized knowledge and advanced skills to generate and customize 3D data, making it difficult for average users to do so. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system that includes a depth sensor for scanning an object, a processing unit for processing the acquired data to generate a three-dimensional model, an input unit for inputting a user's requests for the generated three-dimensional model, a generation unit for analyzing the acquired requests and generating an appropriate data file based on the three-dimensional model, and an output unit for transmitting the generated data file to an output device to generate the object. This system realizes a process that allows users to easily customize and generate various objects without requiring special skills. This allows users to obtain custom items that are perfect for various everyday situations.

[0006] "Depth sensor means" refers to devices and technologies for obtaining depth information of an object, and specifically includes optical depth measurement devices and LiDAR.

[0007] "Processing means" refers to a computing device or software algorithm for analyzing and transforming acquired data and generating a three-dimensional model.

[0008] "Input means" refers to an interface and device that allows a user to input requests and customization information for a three-dimensional model.

[0009] "Generation means" refers to the algorithms and software that adjust the shape and size of the three-dimensional model based on the user's requests and generate an appropriate data file.

[0010] "Output means" refers to a device or system for outputting the generated data file as an actual object, specifically a 3D printer. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2]1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0012] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0013] First, the terms used in the following description will be explained.

[0014] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0015] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0016] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0017] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0018] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0019] [First embodiment]

[0020] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0021] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0022] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0023] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0024] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0025] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0026] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0027] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0028] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0029] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0030] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0031] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0032] The present invention provides a system that allows users to easily create something that fits perfectly into a specific location or object. The system includes a depth sensor, a processing unit, an input unit, a generating unit, and an output unit, and these components work together to generate a custom item based on the user's request.

[0033] First, a user scans an object using a depth sensor (e.g., a LiDAR sensor built into a smartphone). The user holds the smartphone and moves it around the object to acquire depth information. This information is sent in real time to a processing unit, which then generates a 3D model.

[0034] Next, the user uses a chat interface within the application on the device to input customization requests for the generated 3D model. For example, the user can input a specific request such as "I want a cup holder with a diameter of 10 cm and a height of 5 cm." This input information is analyzed and sent to the generation means.

[0035] The generator analyzes the user's requests and adjusts the shape and size of the 3D model data appropriately using a dedicated algorithm, ultimately generating an .obj file that reflects the user's requests.

[0036] The generated .obj file is displayed to the user via the terminal, and the user can confirm its contents. After confirmation, the user sends the generated data file from the terminal to the output means, i.e., the 3D printer, which generates a physical item based on the .obj file.

[0037] As a concrete example, let's say a user wants to create a smartphone holder that fits perfectly inside a car's drink holder. In this case, the user uses the smartphone's LiDAR sensor to scan the inside of the drink holder. Next, in the application, the user enters, "I want to create a smartphone holder that is 7 cm in diameter and 10 cm high." The generator analyzes this and generates an .obj file of a smartphone holder shaped to fit inside the drink holder. Finally, the user can print this file on a 3D printer and use the finished smartphone holder in their car.

[0038] In this way, the system of the present invention provides a means for users to easily create custom items they need in their daily lives, and can be used without any special skills.

[0039] The processing flow will be explained below.

[0040] Step 1:

[0041] User: Launches the dedicated application on the smartphone and starts using the LiDAR sensor by pointing it at the object to be scanned. The user holds the smartphone and moves around the object to acquire depth information.

[0042] Step 2:

[0043] On the device: Depth data acquired by the LiDAR sensor is processed in real time to generate 3D point cloud data, which is then visually displayed in the application, allowing the user to follow the progress of the scan.

[0044] Step 3:

[0045] User: Once the scan is complete, press the "Scan Complete" button within the application to finalize the scan data.

[0046] Step 4:

[0047] Terminal: Compresses and converts the format of the confirmed point cloud data and sends it to a server via the Internet.

[0048] Step 5:

[0049] Server: Receives point cloud data sent from the device, verifies the data's normality and consistency, then generates a 3D mesh from the point cloud data and converts it into a format usable by the generation AI.

[0050] Step 6:

[0051] User: After completing the scan, the user accesses the chat interface within the application and enters their requirements and wishes for the item they want to create. For example, they could enter, "I want to make a smartphone holder with a diameter of 10 cm."

[0052] Step 7:

[0053] Terminal: The text analysis engine processes the user's input, extracts the appropriate instruction information, and sends it to the server.

[0054] Step 8:

[0055] Server: Based on instructions from the text analysis engine, the generation AI adjusts the shape and size of the 3D mesh data and generates an .obj file that meets the requirements.

[0056] Step 9:

[0057] Server: Sends the generated .obj files to the device.

[0058] Step 10:

[0059] Terminal: The received .obj file is displayed to the user, providing a preview, allowing the user to review the details of the result and request changes or corrections if necessary.

[0060] Step 11:

[0061] User: If there is no problem with the .obj file, select the "3D Print" option.

[0062] Step 12:

[0063] Terminal: Convert the .obj file into a format suitable for a 3D printer (e.g. G-code) and send it to the 3D printer.

[0064] Step 13:

[0065] Device: Once the print setup is complete, start your 3D print.

[0066] Step 14:

[0067] User: Once the 3D printer has finished printing, they retrieve the physical item and place or use it where they want.

[0068] These steps allow the system of the present invention to be implemented in a way that allows users to easily create and print custom items that fit perfectly into specific locations or objects.

[0069] Example 1

[0070] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0071] Previously, specialized knowledge and expensive equipment were required for users to create custom items that fit perfectly to specific objects or locations, making it difficult for ordinary users to create custom items they need in their daily lives.

[0072] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0073] In this invention, the server includes: means for scanning the periphery of an object using a depth sensor means and acquiring depth information; processing means for generating a 3D model in real time based on the depth information and transmitting it to the server; input means for providing a chat interface for users to input customization requests; generation means for analyzing the user's customization requests, adjusting the 3D model using a generative AI model, and generating an optimal data file; and output means for returning the generated data file to the terminal, displaying it for the user to confirm, and finally transmitting it to an output device. This enables general users to easily create custom items without special skills or expensive equipment.

[0074] A "depth sensor" is a device for measuring the distance and shape around an object.

[0075] A "three-dimensional model" is digital data that represents the shape and structure of an object or space in three-dimensional space.

[0076] A "server" is a computer system that processes and provides data in response to requests from multiple clients via a network.

[0077] "Processing means" refers to hardware or software for analyzing input data and performing necessary calculations or transformations.

[0078] A "generative AI model" is an artificial intelligence model used to generate and process data based on user instructions.

[0079] An "input means" is a device or software that includes an interface through which a user can input instructions or data into a system.

[0080] A "generator" is a mechanism for generating new data or models based on input data or instructions.

[0081] "Output means" refers to hardware or software for displaying and outputting the generated data and models.

[0082] A "prompt sentence" is a text sentence that a user enters to convey instructions or requests to the system.

[0083] A "3D viewer" is software for visually checking and manipulating 3D models.

[0084] A "3D printer" is a device used to create physical three-dimensional objects based on digital data.

[0085] The present invention provides a system that allows users to easily create custom items that fit perfectly into specific locations or objects. This system generates original items based on the user's requests by using a depth sensor, a processing unit, an input unit, a generating unit, and an output unit that work in cooperation with each other.

[0086] First, the user scans the object using the smartphone's built-in LiDAR sensor (depth sensor means). The user holds the smartphone and moves it around the object to acquire depth information. This depth information is sent in real time to the smartphone's processing means, and a 3D model is generated.

[0087] Next, the user inputs their customization requests for the generated 3D model through a chat interface (input means) provided within the smartphone application. For example, they might input a prompt such as, "I would like to create a smartphone holder with a diameter of 7 cm and a height of 10 cm." This input information is sent to the server (generation means), which analyzes the request using a generative AI model and generates a 3D model adjusted to the appropriate shape and size. The server then generates this 3D model as an .obj file and returns it to the device.

[0088] The user reviews the generated .obj file on their device. Using a 3D viewer, they can check the appearance and dimensions of the model. Once the user is satisfied with the content of the file, they send the .obj file from their device to a 3D printer (output means). The 3D printer generates a physical object based on the .obj file.

[0089] As a concrete example, consider a case where a user wants to create a smartphone holder that fits into a car's drink holder. In this case, the user uses the smartphone's LiDAR sensor to scan the inside of the drink holder. Then, in the application, the user enters, "I want to create a smartphone holder that is 7 cm in diameter and 10 cm high." The server's generator analyzes this request and generates an .obj file of a smartphone holder shaped to fit inside the drink holder. Finally, the user can print this file on a 3D printer and use the completed smartphone holder in their car.

[0090] Examples of prompts:

[0091] "I want to make a cup holder that is 10cm in diameter and 5cm in height."

[0092] "I want to make a seasoning rack that is 12cm wide, 15cm high, and 5cm deep."

[0093] "I want to create a custom case that fits perfectly to a specific shape."

[0094] In this way, the system of the present invention provides a means for users to easily create custom items needed in everyday life without having to have special skills.

[0095] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0096] Step 1:

[0097] The user scans an object using the smartphone's LiDAR sensor. The input is real-time depth information acquired while moving around the object. The depth information is sent to the device's processing means. The device generates a 3D model based on this and temporarily stores it in memory. The user can then check the status of the depth information acquisition in real time on the smartphone screen. The scan progress is displayed on the screen, along with the message "Scan complete."

[0098] Step 2:

[0099] The processing means of the terminal transmits the generated 3D model to the server. The generated 3D model is included as input. The 3D model is compressed and transmitted to the server via the network. The server receives the 3D model and begins analysis. During data transmission, a progress bar and messages are displayed.

[0100] Step 3:

[0101] The user uses the chat interface on the device to input their specific customization request. For example, they might input a prompt such as, "I want to create a smartphone holder that is 7 cm in diameter and 10 cm in height." The input contains the user's customization request. The device parses this prompt and sends it to the server. The input is then displayed in the chat interface, and a confirmation message is displayed.

[0102] Step 4:

[0103] The server uses a generative AI model to generate a custom model based on the received user customization requests and 3D model. The inputs are a prompt from the user and the 3D model. The server uses the generative AI model to execute an internal algorithm and generate a 3D model that matches the user's requests. This generated model is saved in .obj file format. Data processing is performed on the server side, and the progress is displayed.

[0104] Step 5:

[0105] The server sends the generated .obj file to the terminal. The input is the generated .obj file. The file is compressed and sent to the terminal over the network. The terminal receives the .obj file and notifies the user. As a result, the user's terminal displays "Custom model generated."

[0106] Step 6:

[0107] The user uses the terminal to check the generated .obj file. Using the 3D viewer, the appearance and dimensions of the model can be checked in detail. The input is the generated .obj file. The user checks and approves it. In operation, the 3D model is displayed on the screen and the user performs the check operation.

[0108] Step 7:

[0109] The user sends the .obj file from the device to the 3D printer, which is the output device. The input is the .obj file that the user has confirmed. The 3D printer generates the physical item based on this. As a result, the status of the 3D printer is displayed on the device screen, allowing the user to check the progress of the print. For example, the process of the user printing a smartphone holder is displayed in detail.

[0110] By following these steps, users can create custom items without any special expertise.

[0111] (Application example 1)

[0112] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0113] In modern factories, creating custom tools and parts to fit specific machine components is a complex and time-consuming task. Traditional methods require specialized knowledge and make it difficult to efficiently design and manufacture custom parts. This can lead to process interruptions and delays, reducing production efficiency, especially when rapid changes or modifications to production lines are required. To solve this problem, a system is needed that allows users to easily design and quickly manufacture custom parts.

[0114] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0115] In this invention, the server includes a sensor means for scanning an object, an information processing means for processing data acquired from the sensor means to generate a three-dimensional model, an input means for inputting a user's request for the generated three-dimensional model, a generation means for analyzing the request acquired from the input means and generating an appropriate data file based on the three-dimensional model, an output means for transmitting the generated data file to an output device to generate an object, and a control means implemented in a computer for controlling a machine tool appropriate for the generated data file, thereby enabling the rapid and efficient design and manufacturing of required custom tools and parts in a factory.

[0116] "Sensor means" refers to a device used to acquire information about the three-dimensional shape and depth of an object. Optical depth measurement devices and LiDAR sensors are mainly used.

[0117] "Information processing means" refers to a computer system that processes data acquired from the sensor means and generates a three-dimensional model. It analyzes, converts, and generates the data.

[0118] "Input means" refers to the interface through which the user inputs specific requests to the system. Chat interfaces and touch screens are typically used.

[0119] "Generation means" refers to a device or function that analyzes the 3D model and generates an appropriate data file based on the user's request input via the input means. The shape and size of the model are adjusted using an algorithm.

[0120] The "output means" is a device that generates the generated data file as a physical object. 3D printers and machine tools are usually used.

[0121] The "control means" is a system for controlling the machine tool based on the generated data file, and instructs the operation of the machine tool using a computer program.

[0122] Hardware Configuration

[0123] The system includes the following major hardware components:

[0124] 1. Sensor means: Optical depth measurement devices (such as LiDAR sensors) for obtaining three-dimensional shape and depth information of objects.

[0125] 2. Information processing means: A computer that processes the data obtained from the sensor means and generates a three-dimensional model.

[0126] 3. Input means: The interface through which users input their requests (e.g., a chat interface or touch screen on a smartphone).

[0127] 4. Generation means: A processing device that analyzes the three-dimensional model and generates a data file based on the user's request.

[0128] 5. Output means: A 3D printer or machine tool that produces a physical item based on the generated data file.

[0129] 6. Control means: A computer program that controls the machine tool based on the generated data file.

[0130] Software Structure and Algorithms

[0131] The system uses libraries such as open3d to process 3D point cloud data and generate .obj files, and also uses an AI model to analyze user input and generate an appropriate 3D model.

[0132] Operation procedures and data processing

[0133] The server processes the data and generates the custom items as follows:

[0134] 1. Data acquisition: Using sensor means to acquire depth information of the scanned object. For example, scanning a specific part in a factory with a smartphone's LiDAR sensor.

[0135] 2. 3D model generation: The acquired data is sent to the information processing means, and a 3D model is generated using the open3d library.

[0136] 3. Request input: The user uses the chat interface on their smartphone to input a specific request, for example, "I want to create an adapter that fits a pipe with a diameter of 5 cm and a length of 10 cm."

[0137] 4. Request analysis: The generator analyzes the input request and adjusts the 3D model. The shape and size are automatically adjusted using an AI model.

[0138] 5. Data file generation: The final adjusted 3D model is generated as a data file (.obj format).

[0139] 6. Output: The generated data file is sent to a 3D printer or machine tool to create the physical custom item.

[0140] 7. Machine Tool Control: A control means controls the machine tool based on the generated data file to rapidly manufacture precision custom parts.

[0141] Specific examples

[0142] For example, to create an adapter to fit a specific machine part in a factory, a user might do the following:

[0143] 1. Scan the target pipe with your smartphone's LiDAR sensor.

[0144] 2. Type into the chat interface, "I want to generate an adapter that fits a pipe that is 5cm in diameter and 10cm long."

[0145] 3. The server generates a 3D model based on the acquired data and adjusts it to suit your needs.

[0146] 4. The generated .obj file is sent to a 3D printer to produce the physical adapter.

[0147] Prompt Sentence Examples

[0148] "Create an .obj file that generates a custom adapter that fits a pipe that is 5cm in diameter and 10cm long."

[0149] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0150] Step 1:

[0151] A sensor means is used to obtain depth information of the scanned object.

[0152] Input: Object to scan

[0153] Data processing: A depth sensor (LiDAR sensor) scans the area around the object and obtains three-dimensional point cloud data.

[0154] Output: Acquired 3D point cloud data

[0155] Step 2:

[0156] The information processing means processes the acquired three-dimensional point cloud data to generate a three-dimensional model.

[0157] Input: 3D point cloud data

[0158] Data processing: Analyze the 3D point cloud data using the open3d library and generate a 3D model.

[0159] Output: Generated 3D model

[0160] Step 3:

[0161] The user uses the chat interface of the terminal to input a custom request.

[0162] Input: User's custom request (e.g. "I want to create an adapter that fits a pipe with a diameter of 5 cm and a length of 10 cm")

[0163] How it works: A user types a specific request into the chat interface on their smartphone or device.

[0164] Output: Input custom request

[0165] Step 4:

[0166] The generating means analyzes the input request and adjusts the three-dimensional model.

[0167] Input: Generated 3D model, custom request

[0168] Data calculation: Using a generative AI model, the shape and size of the 3D model are adjusted based on the input customization requests. Specifically, the system analyzes the user's requests and appropriately changes the shape parameters of the 3D model.

[0169] Output: Adjusted 3D model

[0170] Step 5:

[0171] Generate a suitable data file (.obj format) based on the adjusted 3D model.

[0172] Input: Calibrated 3D model

[0173] Data Generation: Export the 3D model in .obj file format.

[0174] Output: Generated .obj file

[0175] Step 6:

[0176] An output method is used to send the generated .obj file to a 3D printer or machine tool to manufacture the physical custom item.

[0177] Input: Generated .obj file

[0178] Data Transfer: Send the generated .obj file over the network to the 3D printer or machine tool.

[0179] How it works: The 3D printer deposits filament based on the .obj file to create a physical custom part.

[0180] Output: The generated physical custom item

[0181] Step 7:

[0182] The control means controls the machine tool based on the generated data file to manufacture precision custom parts.

[0183] Input: Generated .obj file

[0184] Control operation: The control program installed in the machine tool drives the machine tool based on the contents of the .obj file to manufacture a custom part with the specified shape.

[0185] Output: Precision-manufactured custom parts

[0186] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0187] The present invention provides a system that allows users to easily create items that perfectly fit into specific locations or objects. The system includes a depth sensor, a processing unit, an input unit, a generating unit, an output unit, and an emotion engine. These components work together to generate custom items based on the user's requests and emotion information.

[0188] First, a user scans an object using a depth sensor (e.g., an optical depth measurement device built into a smartphone). The user holds the smartphone and moves around the object to acquire depth information. This information is sent in real time to a processing unit, which then generates a three-dimensional model.

[0189] Next, the user uses a chat interface within the application on the device to input customization requests for the generated 3D model. For example, a specific request such as "I want a cup holder with a diameter of 10 cm and a height of 5 cm" can be input into the device. At the same time, the emotion engine analyzes the user's facial expressions and voice to obtain emotional information. This input information and emotional information are analyzed and sent to the generation means.

[0190] The generator analyzes the user's requests and emotional information and adjusts the shape and size of the 3D model data appropriately. For example, if the user is excited, bold elements are added to the design, and if the user is relaxed, the design is adjusted to be calm. A dedicated algorithm is used for this adjustment, and finally an .obj file that reflects the user's requests and emotions is generated.

[0191] The generated .obj file is displayed to the user via the terminal, and the user can confirm its contents. After confirmation, the user sends the generated data file from the terminal to the output means, i.e., the 3D printer, which generates a physical item based on the .obj file.

[0192] As a concrete example, let's say a user wants to create a pen holder that fits perfectly on their desk at home. In this case, the user uses the optical depth measurement device on their smartphone to scan the desired area of ​​the desk. Meanwhile, the emotion engine ensures that the user is relaxed. Next, they input in the application, "I want to create a pen holder that is 10cm tall and 5cm wide." The generator analyzes this and generates an .obj file of the pen holder that incorporates emotional information and calm design elements. Finally, the user can print this file on a 3D printer and place the finished pen holder on their desk.

[0193] In this way, the system of the present invention provides a means for users to easily create custom items they need in their daily lives, and can be used without any special skills. Furthermore, the introduction of an emotion engine enables customization according to the user's emotional state, providing a more satisfying user experience.

[0194] The processing flow will be explained below.

[0195] Step 1:

[0196] User: Launches the dedicated application on the smartphone and starts using the LiDAR sensor by pointing it at the object to be scanned. The user holds the smartphone and moves around the object to acquire depth information.

[0197] Step 2:

[0198] On the device: Depth data acquired by the LiDAR sensor is processed in real time to generate 3D point cloud data, which is then visually displayed in the application, allowing the user to follow the progress of the scan.

[0199] Step 3:

[0200] User: Once the scan is complete, press the "Scan Complete" button within the application to finalize the scan data.

[0201] Step 4:

[0202] Terminal: Compresses and converts the format of the confirmed point cloud data and sends it to a server via the Internet.

[0203] Step 5:

[0204] Server: Receives point cloud data sent from the device, verifies the data's normality and consistency, then generates a 3D mesh from the point cloud data and converts it into a format usable by the generation AI.

[0205] Step 6:

[0206] User: After completing the scan, the user accesses the chat interface within the application and enters their requirements and wishes for the item they wish to create, for example, "I would like to create a vase that is 10cm tall and 7cm in diameter."

[0207] Step 7:

[0208] Terminal: The text analysis engine processes the user's input, extracts the appropriate instruction information, and sends it to the server.

[0209] Step 8:

[0210] Server: Based on instructions from the text analysis engine, the generation AI adjusts the shape and size of the 3D mesh data and generates an .obj file that meets the requirements.

[0211] Step 9:

[0212] Server: Sends the generated .obj files to the device.

[0213] Step 10:

[0214] Terminal: The received .obj file is displayed to the user, providing a preview, allowing the user to review the details of the result and request changes or corrections if necessary.

[0215] Step 11:

[0216] User: If there is no problem with the .obj file, select the "3D Print" option.

[0217] Step 12:

[0218] Terminal: Convert the .obj file into a format suitable for a 3D printer (e.g. G-code) and send it to the 3D printer.

[0219] Step 13:

[0220] Device: Once the print setup is complete, start your 3D print.

[0221] Step 14:

[0222] User: Once the 3D printer has finished printing, they retrieve the physical item and place or use it where they want.

[0223] Step 15:

[0224] On the device: At the same time, the emotion engine acquires emotion information through facial expression recognition and voice analysis, which is then analyzed together with the input information in the chat interface.

[0225] Step 16:

[0226] Server: Reflects the user's emotional information obtained by the emotion engine and adjusts the shape and design when generating the data file. For example, if the user is relaxed, a calm design is suggested.

[0227] Step 17:

[0228] Terminal: The adjusted .obj file is displayed to the user again for final confirmation.

[0229] Step 18:

[0230] User: Once final confirmation is complete, send it to the 3D printer for final printing.

[0231] Through these steps, the system of the present invention is implemented to enable users to easily create and output custom items that fit perfectly into specific locations or objects, and can reflect the user's emotional information in the process.

[0232] Example 2

[0233] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0234] Conventional 3D model generation systems often require users to have detailed technical knowledge and skills to create custom items that fit specific shapes and sizes. Furthermore, they lack the ability to customize designs based on the user's emotional state, leaving the user experience open to improvement.

[0235] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0236] In this invention, the server includes depth sensor means for scanning an object, processing means for processing data acquired from the depth sensor means to generate a three-dimensional model, an emotion engine for analyzing a user's facial expressions and voice to acquire emotion information, and output means for transmitting the generated data file to an output device and generating an object. This allows users to easily create custom items without special skills and enables customization of designs based on emotion information.

[0237] "Depth sensor"

[0238] is a device for measuring the distance and shape of an object, and primarily uses optical technology to obtain depth information.

[0239] "Processing means"

[0240] is a computing device or software that uses acquired data to generate a three-dimensional model.

[0241] "Input means"

[0242] is an interface through which the user inputs customization requests, and through which the user's instructions are transmitted to the system.

[0243] "Emotion Engine"

[0244] is a technology or device for acquiring emotional information by analyzing a user's facial expressions and voice.

[0245] "Generation means"

[0246] is a technology or device that analyzes requests obtained from an input means and emotional information obtained from an emotion engine, and generates an appropriate data file based on a three-dimensional model.

[0247] Output means

[0248] is a device for sending the generated data file to an output device to generate a physical object.

[0249] "Three-dimensional model"

[0250] is a digital representation created based on the shape and size of an object, and refers to three-dimensional structure data composed of depth information.

[0251] "Emotional information"

[0252] This is psychological state data that is analyzed by the emotion engine from the user's facial expressions and voice.

[0253] Data File

[0254] is a file format in which the three-dimensional model data is output after being appropriately processed by the generating means.

[0255] This invention provides a system that allows users to easily create items that perfectly fit into specific locations or objects. The system includes a depth sensor, processing means, input means, an emotion engine, generation means, and output means. These components work together to generate custom items based on the user's requests and emotion information.

[0256] First, the user scans the object using a depth sensor (for example, an optical depth measurement device built into a smartphone). The user holds the smartphone and moves around the object, acquiring depth information. This information is sent in real time to a processing means, which generates a 3D model. Specifically, the smartphone application analyzes the depth data and updates the 3D model every 0.1 seconds. This model is displayed on the application's main screen, where the user can rotate and zoom the model.

[0257] Next, the user uses a chat interface within the application on the device to input customization requests for the generated 3D model. For example, a specific request such as "I want a cup holder with a diameter of 10 cm and a height of 5 cm" can be input into the device. At the same time, the emotion engine analyzes the user's facial expressions and voice to obtain emotional information. This analysis is performed using a camera and microphone. If the user is smiling or speaking in a calm voice, the emotion engine determines that the user is relaxed.

[0258] The desire and emotion information acquired from the input means is sent to the generation means. The generation means analyzes this information and adjusts the 3D model data to an appropriate shape and size. For example, if the user is relaxed, adjustments are made to incorporate calm design elements. This ultimately generates an .obj file that reflects the user's desire and emotion. A Python-based shape generation algorithm is used for this adjustment.

[0259] The generated .obj file is displayed to the user through the terminal, and the user can confirm its contents. After confirming, the user sends the generated data file from the terminal to the output means, i.e., the 3D printer. The 3D printer generates a physical item based on the .obj file.

[0260] As a concrete example, let's say a user wants to create a pen holder that fits perfectly on their desk at home. In this case, the user uses the optical depth measurement device on their smartphone to scan the desired area of ​​the desk. Meanwhile, the emotion engine confirms that the user is relaxed. Next, they input in the application, "I want to create a pen holder that is 10 cm tall and 5 cm wide." The generator analyzes this and generates an .obj file of the pen holder that incorporates calming design elements, incorporating the emotion information. Finally, the user can print this file on a 3D printer and place the finished pen holder on their desk.

[0261] An example prompt is:

[0262] "I would like to make a pen holder that is 10cm tall and 5cm wide. Please keep the design calm."

[0263] In this way, the system of the present invention provides a means for users to easily create custom items they need in their daily lives, and can be used without any special skills. Furthermore, the introduction of an emotion engine enables customization according to the user's emotional state, providing a more satisfying user experience.

[0264] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0265] A detailed explanation of the system's program divided into processing steps

[0266] Step 1: Acquiring and sending depth information

[0267] concrete action

[0268] The user launches the smartphone app and switches to scan mode. The device uses the smartphone's built-in optical depth measurement device to scan the area around an object designated by the user and collect depth information.

[0269] input

[0270] The user selects the object they want to scan and moves around while holding the smartphone.

[0271] Data processing and calculation

[0272] The device processes the acquired depth data in real time and generates three-dimensional coordinate data.

[0273] output

[0274] The processed three-dimensional coordinate data is generated and sent to the next step.

[0275] Step 2: Generate a 3D model

[0276] concrete action

[0277] The device receives the processed depth data in real time and generates a 3D model, which the user can view on the application screen and manipulate, such as rotating and zooming.

[0278] input

[0279] Three-dimensional coordinate data obtained by scanning.

[0280] Data processing and calculation

[0281] The processing means builds a model based on the 3D coordinate data, and updates the model every 0.1 seconds.

[0282] output

[0283] The generated three-dimensional model is displayed in real time on the user's terminal screen.

[0284] Step 3: Enter your customization request

[0285] concrete action

[0286] The user uses the chat interface within the application on the device to input customization requests for the generated 3D model. For example, the user might input, "I want to create a cup holder that is 10 cm in diameter and 5 cm in height."

[0287] input

[0288] Specific customization requests entered by the user into the chat interface.

[0289] Data processing and calculation

[0290] The user's request is analyzed as text and passed to the generating means as an appropriate parameter.

[0291] output

[0292] The desired parameters after text analysis are sent to the next step.

[0293] Step 4: Acquiring emotional information

[0294] concrete action

[0295] Using a camera and microphone, the emotion engine analyzes the user's facial expressions and tone of voice. For example, if the user is smiling or speaking in a calm voice, it determines that the user is relaxed.

[0296] input

[0297] The user's facial expression data and voice data.

[0298] Data processing and calculation

[0299] The emotion engine uses facial recognition and voice analysis algorithms to determine the user's emotional state.

[0300] output

[0301] The analyzed emotion information is sent to the next step.

[0302] Step 5: Analyze and adjust the information

[0303] concrete action

[0304] The terminal receives the user's customization request and emotional information, and the generation means analyzes this information and adjusts the 3D model appropriately. For example, if the user is relaxed, an .obj file incorporating calm design elements is generated.

[0305] input

[0306] User customization request parameters and emotional information.

[0307] Data processing and calculation

[0308] The generating means adjusts the shape and size of the three-dimensional model based on the shape generation algorithm and generates an .obj file.

[0309] output

[0310] The generated .obj file is sent to the next step.

[0311] Step 6: Check and output the results

[0312] concrete action

[0313] The generated .obj file is displayed on the terminal screen for the user to review. Once the user confirms the design, the file is sent to the 3D printer, which creates the physical item based on the .obj file.

[0314] input

[0315] The generated .obj file.

[0316] Data processing and calculation

[0317] The system uses file transfer protocol to send the .obj file to the 3D printer.

[0318] output

[0319] The physical item is produced by a 3D printer.

[0320] In this way, each processing step has specific inputs and outputs, and specific actions are taken to make the entire system function.

[0321] (Application example 2)

[0322] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0323] Modern factory manufacturing processes require the rapid creation of specialized tools and parts. However, traditional manufacturing methods require significant time and effort to design and create specialized tools, significantly reducing the efficiency of factory workers. Furthermore, existing systems struggle to automatically generate custom tools that reflect user feedback and emotions. As a result, the flexibility and efficiency of the manufacturing process are limited. To address these challenges, a new system is needed that can generate custom tools based on user requests and emotion information in a fast and automated manner.

[0324] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a depth sensor means for scanning an object, a processing means for processing data acquired from the depth sensor means to generate a three-dimensional model, an input means for inputting a user's desires and emotional information for the generated three-dimensional model, an output means for transmitting the generated data file to an output device and generating an object, and an industrial robot means for generating an automatically adjusted custom tool based on the analyzed desires and emotional information. This enables factory workers to quickly and efficiently create specialized tools. Furthermore, providing custom tools that reflect the user's emotional information significantly improves the flexibility and efficiency of the manufacturing process.

[0325] The "depth sensor means" is a device for detecting the depth and distance information of an object.

[0326] The "processing means" refers to a device or program for generating a three-dimensional model based on the acquired depth data.

[0327] "Input means" refers to a device or interface that allows a user to input requests and emotional information to a three-dimensional model.

[0328] "Generation means" refers to a device or program that analyzes the user's requests and emotional information and generates a data file based on a three-dimensional model.

[0329] An "output means" is a device or interface for converting the generated data file into a physical form.

[0330] An "industrial robotic tool" is a device such as a robotic arm for automatically creating custom tools or objects based on a generated data file.

[0331] "User requirements" are specific requests that users have for the system, such as the shape, size, and functionality of custom tools or objects.

[0332] "Emotion information" is data that indicates the emotional state of the user, obtained from facial expressions, voice, etc.

[0333] A "3D model" is three-dimensional digital data that represents the depth and shape of a scanned object.

[0334] The "data file" is the design data of a custom tool based on a three-dimensional model or user requirements, which is created by the generating means.

[0335] "Custom Tools" are specialized tools and parts designed and produced based on a user's specific requirements.

[0336] To implement this invention, a system is constructed in the following steps: First, a user scans an object using a depth sensor, which can be a smartphone or tablet equipped with an optical depth measurement device, and data on the object's depth and shape is acquired.

[0337] This depth data is then sent in real time to a processing unit within the device, which generates a three-dimensional model using 3D modeling software such as Blender. Users can also use input units within the smartphone application to enter requests. For example, they can use a chat interface to enter a specific request, such as "I want a wrench with a diameter of 5 cm."

[0338] Furthermore, the user's emotional information is acquired through the device's camera and microphone and analyzed by an emotion engine (e.g., Microsoft Azure Face API), allowing customization according to the user's emotional state.

[0339] The generation means adjusts the shape and size of the 3D model data appropriately based on the user's requests and emotional information. A dedicated algorithm analyzes this, and finally generates a data file in .obj file format. This data file is sent by the server to a robot arm (e.g., Universal Robots' UR5). The server instructs the robot arm to create a custom tool based on the generated data file.

[0340] Finally, the robotic arm creates a custom tool based on the generated .obj file, which is then physically printed using a 3D printer (e.g., Formlabs Form 3) as the output method, allowing users to quickly obtain the specialized tool they desire.

[0341] As a concrete example, imagine a factory worker needs a specific tool that perfectly fits the design of a new product line. In this case, the worker uses their smartphone to scan an existing part, and the emotion engine detects their emotional state as relaxed. They then type, "I need a special wrench that's 10cm tall and 5cm wide" into the application's chat interface. The generator analyzes this, automatically adjusts the design of the custom tool based on the emotion information, and generates an .obj file. A robotic arm then creates the tool based on this design, and finally, a 3D printer physically prints it. The worker can then use the tool immediately.

[0342] Here is an example of a prompt to input to a generative AI model:

[0343] Based on the invention below, design an application for a custom tool-making system for factory workers.

[0344] Scan objects using depth sensing means

[0345] The acquired depth data is processed in real time to generate a 3D model

[0346] Enter user requests using the chat interface

[0347] Acquire and analyze user emotional information using an emotion engine

[0348] Generate .obj files based on 3D model data

[0349] The generated data file is sent to the robot arm

[0350] Robotic arm automatically generates custom tools

[0351] Finally, the custom tool is printed on a 3D printer.

[0352] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0353] Step 1:

[0354] The user scans the object using the depth sensor means. The depth sensor means uses an optical depth measurement device on a smartphone or tablet to acquire data on the depth and shape of the object. The input is the object to be scanned, and the output is the depth data of the object. The user holds the smartphone and moves around the object to be scanned.

[0355] Step 2:

[0356] The acquired depth data is sent in real time to the processing means within the device. As the processing means, 3D modeling software (e.g. Blender) is used to generate a three-dimensional model based on the depth data. The input is depth data and the output is a three-dimensional model. The device analyzes the depth data and generates a three-dimensional model of the object.

[0357] Step 3:

[0358] The user uses a chat interface within the smartphone application to input their requests for the generated 3D model. Text input or voice input can be used as the input method. The input is the user's request (e.g., "I want a wrench with a diameter of 5 cm"), and the output is the request information. The user inputs a specific request.

[0359] Step 4:

[0360] The device acquires the user's emotional information through the camera and microphone and analyzes it using an emotion engine (e.g., Microsoft Azure Face API). The input is the user's facial expression and voice, and the output is the analyzed emotional information. The device determines the user's emotional state.

[0361] Step 5:

[0362] The generator adjusts the shape and size of the 3D model data appropriately based on the user's request and emotional information. A dedicated algorithm analyzes this and finally generates a data file in .obj file format. The input is request information and emotional information, and the output is the adjusted 3D model data and .obj file. The generator automatically adjusts the design.

[0363] Step 6:

[0364] The server sends the generated data file to an industrial robotic means (e.g., a robot arm). The input is an .obj file, and the output is the data file transfer to the robot arm. The server sends instructions to the robot arm to process the generated model.

[0365] Step 7:

[0366] The robot arm creates a custom tool based on the submitted .obj file. The input is the .obj file, and the output is the physical custom tool. The robot arm then machines the tool into the specified shape.

[0367] Step 8:

[0368] Optionally, the custom tool is physically printed with a 3D printer as the output medium. The input is an .obj file and the output is a 3D printed custom tool. The 3D printer generates the physical tool.

[0369] The above process provides a system that allows users to quickly and efficiently create specialized tools.

[0370] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0371] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0372] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0373] [Second embodiment]

[0374] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0375] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0376] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0377] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0378] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0379] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0380] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0381] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0382] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0383] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0384] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0385] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0386] The present invention provides a system that allows users to easily create something that fits perfectly into a specific location or object. The system includes a depth sensor, a processing unit, an input unit, a generating unit, and an output unit, and these components work together to generate a custom item based on the user's request.

[0387] First, a user scans an object using a depth sensor (e.g., a LiDAR sensor built into a smartphone). The user holds the smartphone and moves it around the object to acquire depth information. This information is sent in real time to a processing unit, which then generates a 3D model.

[0388] Next, the user uses a chat interface within the application on the device to input customization requests for the generated 3D model. For example, the user can input a specific request such as "I want a cup holder with a diameter of 10 cm and a height of 5 cm." This input information is analyzed and sent to the generation means.

[0389] The generator analyzes the user's requests and adjusts the shape and size of the 3D model data appropriately using a dedicated algorithm, ultimately generating an .obj file that reflects the user's requests.

[0390] The generated .obj file is displayed to the user via the terminal, and the user can confirm its contents. After confirmation, the user sends the generated data file from the terminal to the output means, i.e., the 3D printer, which generates a physical item based on the .obj file.

[0391] As a concrete example, let's say a user wants to create a smartphone holder that fits perfectly inside a car's drink holder. In this case, the user uses the smartphone's LiDAR sensor to scan the inside of the drink holder. Next, in the application, the user enters, "I want to create a smartphone holder that is 7 cm in diameter and 10 cm high." The generator analyzes this and generates an .obj file of a smartphone holder shaped to fit inside the drink holder. Finally, the user can print this file on a 3D printer and use the finished smartphone holder in their car.

[0392] In this way, the system of the present invention provides a means for users to easily create custom items they need in their daily lives, and can be used without any special skills.

[0393] The processing flow will be explained below.

[0394] Step 1:

[0395] User: Launches the dedicated application on the smartphone and starts using the LiDAR sensor by pointing it at the object to be scanned. The user holds the smartphone and moves around the object to acquire depth information.

[0396] Step 2:

[0397] On the device: Depth data acquired by the LiDAR sensor is processed in real time to generate 3D point cloud data, which is then visually displayed in the application, allowing the user to follow the progress of the scan.

[0398] Step 3:

[0399] User: Once the scan is complete, press the "Scan Complete" button within the application to finalize the scan data.

[0400] Step 4:

[0401] Terminal: Compresses and converts the format of the confirmed point cloud data and sends it to a server via the Internet.

[0402] Step 5:

[0403] Server: Receives point cloud data sent from the device, verifies the data's normality and consistency, then generates a 3D mesh from the point cloud data and converts it into a format usable by the generation AI.

[0404] Step 6:

[0405] User: After completing the scan, the user accesses the chat interface within the application and enters their requirements and wishes for the item they want to create. For example, they could enter, "I want to make a smartphone holder with a diameter of 10 cm."

[0406] Step 7:

[0407] Terminal: The text analysis engine processes the user's input, extracts the appropriate instruction information, and sends it to the server.

[0408] Step 8:

[0409] Server: Based on instructions from the text analysis engine, the generation AI adjusts the shape and size of the 3D mesh data and generates an .obj file that meets the requirements.

[0410] Step 9:

[0411] Server: Sends the generated .obj files to the device.

[0412] Step 10:

[0413] Terminal: The received .obj file is displayed to the user, providing a preview, allowing the user to review the details of the result and request changes or corrections if necessary.

[0414] Step 11:

[0415] User: If there is no problem with the .obj file, select the "3D Print" option.

[0416] Step 12:

[0417] Terminal: Convert the .obj file into a format suitable for a 3D printer (e.g. G-code) and send it to the 3D printer.

[0418] Step 13:

[0419] Device: Once the print setup is complete, start your 3D print.

[0420] Step 14:

[0421] User: Once the 3D printer has finished printing, they retrieve the physical item and place or use it where they want.

[0422] These steps allow the system of the present invention to be implemented in a way that allows users to easily create and print custom items that fit perfectly into specific locations or objects.

[0423] Example 1

[0424] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0425] Previously, specialized knowledge and expensive equipment were required for users to create custom items that fit perfectly to specific objects or locations, making it difficult for ordinary users to create custom items they need in their daily lives.

[0426] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0427] In this invention, the server includes: means for scanning the periphery of an object using a depth sensor means and acquiring depth information; processing means for generating a 3D model in real time based on the depth information and transmitting it to the server; input means for providing a chat interface for users to input customization requests; generation means for analyzing the user's customization requests, adjusting the 3D model using a generative AI model, and generating an optimal data file; and output means for returning the generated data file to the terminal, displaying it for the user to confirm, and finally transmitting it to an output device. This enables general users to easily create custom items without special skills or expensive equipment.

[0428] A "depth sensor" is a device for measuring the distance and shape around an object.

[0429] A "three-dimensional model" is digital data that represents the shape and structure of an object or space in three-dimensional space.

[0430] A "server" is a computer system that processes and provides data in response to requests from multiple clients via a network.

[0431] "Processing means" refers to hardware or software for analyzing input data and performing necessary calculations or transformations.

[0432] A "generative AI model" is an artificial intelligence model used to generate and process data based on user instructions.

[0433] An "input means" is a device or software that includes an interface through which a user can input instructions or data into a system.

[0434] A "generator" is a mechanism for generating new data or models based on input data or instructions.

[0435] "Output means" refers to hardware or software for displaying and outputting the generated data and models.

[0436] A "prompt sentence" is a text sentence that a user enters to convey instructions or requests to the system.

[0437] A "3D viewer" is software for visually checking and manipulating 3D models.

[0438] A "3D printer" is a device used to create physical three-dimensional objects based on digital data.

[0439] The present invention provides a system that allows users to easily create custom items that fit perfectly into specific locations or objects. This system generates original items based on the user's requests by using a depth sensor, a processing unit, an input unit, a generating unit, and an output unit that work in cooperation with each other.

[0440] First, the user scans the object using the smartphone's built-in LiDAR sensor (depth sensor means). The user holds the smartphone and moves it around the object to acquire depth information. This depth information is sent in real time to the smartphone's processing means, and a 3D model is generated.

[0441] Next, the user inputs their customization requests for the generated 3D model through a chat interface (input means) provided within the smartphone application. For example, they might input a prompt such as, "I would like to create a smartphone holder with a diameter of 7 cm and a height of 10 cm." This input information is sent to the server (generation means), which analyzes the request using a generative AI model and generates a 3D model adjusted to the appropriate shape and size. The server then generates this 3D model as an .obj file and returns it to the device.

[0442] The user reviews the generated .obj file on their device. Using a 3D viewer, they can check the appearance and dimensions of the model. Once the user is satisfied with the content of the file, they send the .obj file from their device to a 3D printer (output means). The 3D printer generates a physical object based on the .obj file.

[0443] As a concrete example, consider a case where a user wants to create a smartphone holder that fits into a car's drink holder. In this case, the user uses the smartphone's LiDAR sensor to scan the inside of the drink holder. Then, in the application, the user enters, "I want to create a smartphone holder that is 7 cm in diameter and 10 cm high." The server's generator analyzes this request and generates an .obj file of a smartphone holder shaped to fit inside the drink holder. Finally, the user can print this file on a 3D printer and use the completed smartphone holder in their car.

[0444] Examples of prompts:

[0445] "I want to make a cup holder that is 10cm in diameter and 5cm in height."

[0446] "I want to make a seasoning rack that is 12cm wide, 15cm high, and 5cm deep."

[0447] "I want to create a custom case that fits perfectly to a specific shape."

[0448] In this way, the system of the present invention provides a means for users to easily create custom items needed in everyday life without having to have special skills.

[0449] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0450] Step 1:

[0451] The user scans an object using the smartphone's LiDAR sensor. The input is real-time depth information acquired while moving around the object. The depth information is sent to the device's processing means. The device generates a 3D model based on this and temporarily stores it in memory. The user can then check the status of the depth information acquisition in real time on the smartphone screen. The scan progress is displayed on the screen, along with the message "Scan complete."

[0452] Step 2:

[0453] The processing means of the terminal transmits the generated 3D model to the server. The generated 3D model is included as input. The 3D model is compressed and transmitted to the server via the network. The server receives the 3D model and begins analysis. During data transmission, a progress bar and messages are displayed.

[0454] Step 3:

[0455] The user uses the chat interface on the device to input their specific customization request. For example, they might input a prompt such as, "I want to create a smartphone holder that is 7 cm in diameter and 10 cm in height." The input contains the user's customization request. The device parses this prompt and sends it to the server. The input is then displayed in the chat interface, and a confirmation message is displayed.

[0456] Step 4:

[0457] The server uses a generative AI model to generate a custom model based on the received user customization requests and 3D model. The inputs are a prompt from the user and the 3D model. The server uses the generative AI model to execute an internal algorithm and generate a 3D model that matches the user's requests. This generated model is saved in .obj file format. Data processing is performed on the server side, and the progress is displayed.

[0458] Step 5:

[0459] The server sends the generated .obj file to the terminal. The input is the generated .obj file. The file is compressed and sent to the terminal over the network. The terminal receives the .obj file and notifies the user. As a result, the user's terminal displays "Custom model generated."

[0460] Step 6:

[0461] The user uses the terminal to check the generated .obj file. Using the 3D viewer, the appearance and dimensions of the model can be checked in detail. The input is the generated .obj file. The user checks and approves it. In operation, the 3D model is displayed on the screen and the user performs the check operation.

[0462] Step 7:

[0463] The user sends the .obj file from the device to the 3D printer, which is the output device. The input is the .obj file that the user has confirmed. The 3D printer generates the physical item based on this. As a result, the status of the 3D printer is displayed on the device screen, allowing the user to check the progress of the print. For example, the process of the user printing a smartphone holder is displayed in detail.

[0464] By following these steps, users can create custom items without any special expertise.

[0465] (Application example 1)

[0466] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0467] In modern factories, creating custom tools and parts to fit specific machine components is a complex and time-consuming task. Traditional methods require specialized knowledge and make it difficult to efficiently design and manufacture custom parts. This can lead to process interruptions and delays, reducing production efficiency, especially when rapid changes or modifications to production lines are required. To solve this problem, a system is needed that allows users to easily design and quickly manufacture custom parts.

[0468] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0469] In this invention, the server includes a sensor means for scanning an object, an information processing means for processing data acquired from the sensor means to generate a three-dimensional model, an input means for inputting a user's request for the generated three-dimensional model, a generation means for analyzing the request acquired from the input means and generating an appropriate data file based on the three-dimensional model, an output means for transmitting the generated data file to an output device to generate an object, and a control means implemented in a computer for controlling a machine tool appropriate for the generated data file, thereby enabling the rapid and efficient design and manufacturing of required custom tools and parts in a factory.

[0470] "Sensor means" refers to a device used to acquire information about the three-dimensional shape and depth of an object. Optical depth measurement devices and LiDAR sensors are mainly used.

[0471] "Information processing means" refers to a computer system that processes data acquired from the sensor means and generates a three-dimensional model. It analyzes, converts, and generates the data.

[0472] "Input means" refers to the interface through which the user inputs specific requests to the system. Chat interfaces and touch screens are typically used.

[0473] "Generation means" refers to a device or function that analyzes the 3D model and generates an appropriate data file based on the user's request input via the input means. The shape and size of the model are adjusted using an algorithm.

[0474] The "output means" is a device that generates the generated data file as a physical object. 3D printers and machine tools are usually used.

[0475] The "control means" is a system for controlling the machine tool based on the generated data file, and instructs the operation of the machine tool using a computer program.

[0476] Hardware Configuration

[0477] The system includes the following major hardware components:

[0478] 1. Sensor means: Optical depth measurement devices (such as LiDAR sensors) for obtaining three-dimensional shape and depth information of objects.

[0479] 2. Information processing means: A computer that processes the data obtained from the sensor means and generates a three-dimensional model.

[0480] 3. Input means: The interface through which users input their requests (e.g., a chat interface or touch screen on a smartphone).

[0481] 4. Generation means: A processing device that analyzes the three-dimensional model and generates a data file based on the user's request.

[0482] 5. Output means: A 3D printer or machine tool that produces a physical item based on the generated data file.

[0483] 6. Control means: A computer program that controls the machine tool based on the generated data file.

[0484] Software Structure and Algorithms

[0485] The system uses libraries such as open3d to process 3D point cloud data and generate .obj files, and also uses an AI model to analyze user input and generate an appropriate 3D model.

[0486] Operation procedures and data processing

[0487] The server processes the data and generates the custom items as follows:

[0488] 1. Data acquisition: Using sensor means to acquire depth information of the scanned object. For example, scanning a specific part in a factory with a smartphone's LiDAR sensor.

[0489] 2. 3D model generation: The acquired data is sent to the information processing means, and a 3D model is generated using the open3d library.

[0490] 3. Request input: The user uses the chat interface on their smartphone to input a specific request, for example, "I want to create an adapter that fits a pipe with a diameter of 5 cm and a length of 10 cm."

[0491] 4. Request analysis: The generator analyzes the input request and adjusts the 3D model. The shape and size are automatically adjusted using an AI model.

[0492] 5. Data file generation: The final adjusted 3D model is generated as a data file (.obj format).

[0493] 6. Output: The generated data file is sent to a 3D printer or machine tool to create the physical custom item.

[0494] 7. Machine Tool Control: A control means controls the machine tool based on the generated data file to rapidly manufacture precision custom parts.

[0495] Specific examples

[0496] For example, to create an adapter to fit a specific machine part in a factory, a user might do the following:

[0497] 1. Scan the target pipe with your smartphone's LiDAR sensor.

[0498] 2. Type into the chat interface, "I want to generate an adapter that fits a pipe that is 5cm in diameter and 10cm long."

[0499] 3. The server generates a 3D model based on the acquired data and adjusts it to suit your needs.

[0500] 4. The generated .obj file is sent to a 3D printer to produce the physical adapter.

[0501] Prompt Sentence Examples

[0502] "Create an .obj file that generates a custom adapter that fits a pipe that is 5cm in diameter and 10cm long."

[0503] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0504] Step 1:

[0505] A sensor means is used to obtain depth information of the scanned object.

[0506] Input: Object to scan

[0507] Data processing: A depth sensor (LiDAR sensor) scans the area around the object and obtains three-dimensional point cloud data.

[0508] Output: Acquired 3D point cloud data

[0509] Step 2:

[0510] The information processing means processes the acquired three-dimensional point cloud data to generate a three-dimensional model.

[0511] Input: 3D point cloud data

[0512] Data processing: Analyze the 3D point cloud data using the open3d library and generate a 3D model.

[0513] Output: Generated 3D model

[0514] Step 3:

[0515] The user uses the chat interface of the terminal to input a custom request.

[0516] Input: User's custom request (e.g. "I want to create an adapter that fits a pipe with a diameter of 5 cm and a length of 10 cm")

[0517] How it works: A user types a specific request into the chat interface on their smartphone or device.

[0518] Output: Input custom request

[0519] Step 4:

[0520] The generating means analyzes the input request and adjusts the three-dimensional model.

[0521] Input: Generated 3D model, custom request

[0522] Data calculation: Using a generative AI model, the shape and size of the 3D model are adjusted based on the input customization requests. Specifically, the system analyzes the user's requests and appropriately changes the shape parameters of the 3D model.

[0523] Output: Adjusted 3D model

[0524] Step 5:

[0525] Generate a suitable data file (.obj format) based on the adjusted 3D model.

[0526] Input: Calibrated 3D model

[0527] Data Generation: Export the 3D model in .obj file format.

[0528] Output: Generated .obj file

[0529] Step 6:

[0530] An output method is used to send the generated .obj file to a 3D printer or machine tool to manufacture the physical custom item.

[0531] Input: Generated .obj file

[0532] Data Transfer: Send the generated .obj file over the network to the 3D printer or machine tool.

[0533] How it works: The 3D printer deposits filament based on the .obj file to create a physical custom part.

[0534] Output: The generated physical custom item

[0535] Step 7:

[0536] The control means controls the machine tool based on the generated data file to manufacture precision custom parts.

[0537] Input: Generated .obj file

[0538] Control operation: The control program installed in the machine tool drives the machine tool based on the contents of the .obj file to manufacture a custom part with the specified shape.

[0539] Output: Precision-manufactured custom parts

[0540] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0541] The present invention provides a system that allows users to easily create items that perfectly fit into specific locations or objects. The system includes a depth sensor, a processing unit, an input unit, a generating unit, an output unit, and an emotion engine. These components work together to generate custom items based on the user's requests and emotion information.

[0542] First, a user scans an object using a depth sensor (e.g., an optical depth measurement device built into a smartphone). The user holds the smartphone and moves around the object to acquire depth information. This information is sent in real time to a processing unit, which then generates a three-dimensional model.

[0543] Next, the user uses a chat interface within the application on the device to input customization requests for the generated 3D model. For example, a specific request such as "I want a cup holder with a diameter of 10 cm and a height of 5 cm" can be input into the device. At the same time, the emotion engine analyzes the user's facial expressions and voice to obtain emotional information. This input information and emotional information are analyzed and sent to the generation means.

[0544] The generator analyzes the user's requests and emotional information and adjusts the shape and size of the 3D model data appropriately. For example, if the user is excited, bold elements are added to the design, and if the user is relaxed, the design is adjusted to be calm. A dedicated algorithm is used for this adjustment, and finally an .obj file that reflects the user's requests and emotions is generated.

[0545] The generated .obj file is displayed to the user via the terminal, and the user can confirm its contents. After confirmation, the user sends the generated data file from the terminal to the output means, i.e., the 3D printer, which generates a physical item based on the .obj file.

[0546] As a concrete example, let's say a user wants to create a pen holder that fits perfectly on their desk at home. In this case, the user uses the optical depth measurement device on their smartphone to scan the desired area of ​​the desk. Meanwhile, the emotion engine ensures that the user is relaxed. Next, they input in the application, "I want to create a pen holder that is 10cm tall and 5cm wide." The generator analyzes this and generates an .obj file of the pen holder that incorporates emotional information and calm design elements. Finally, the user can print this file on a 3D printer and place the finished pen holder on their desk.

[0547] In this way, the system of the present invention provides a means for users to easily create custom items they need in their daily lives, and can be used without any special skills. Furthermore, the introduction of an emotion engine enables customization according to the user's emotional state, providing a more satisfying user experience.

[0548] The processing flow will be explained below.

[0549] Step 1:

[0550] User: Launches the dedicated application on the smartphone and starts using the LiDAR sensor by pointing it at the object to be scanned. The user holds the smartphone and moves around the object to acquire depth information.

[0551] Step 2:

[0552] On the device: Depth data acquired by the LiDAR sensor is processed in real time to generate 3D point cloud data, which is then visually displayed in the application, allowing the user to follow the progress of the scan.

[0553] Step 3:

[0554] User: Once the scan is complete, press the "Scan Complete" button within the application to finalize the scan data.

[0555] Step 4:

[0556] Terminal: Compresses and converts the format of the confirmed point cloud data and sends it to a server via the Internet.

[0557] Step 5:

[0558] Server: Receives point cloud data sent from the device, verifies the data's normality and consistency, then generates a 3D mesh from the point cloud data and converts it into a format usable by the generation AI.

[0559] Step 6:

[0560] User: After completing the scan, the user accesses the chat interface within the application and enters their requirements and wishes for the item they wish to create, for example, "I would like to create a vase that is 10cm tall and 7cm in diameter."

[0561] Step 7:

[0562] Terminal: The text analysis engine processes the user's input, extracts the appropriate instruction information, and sends it to the server.

[0563] Step 8:

[0564] Server: Based on instructions from the text analysis engine, the generation AI adjusts the shape and size of the 3D mesh data and generates an .obj file that meets the requirements.

[0565] Step 9:

[0566] Server: Sends the generated .obj files to the device.

[0567] Step 10:

[0568] Terminal: The received .obj file is displayed to the user, providing a preview, allowing the user to review the details of the result and request changes or corrections if necessary.

[0569] Step 11:

[0570] User: If there is no problem with the .obj file, select the "3D Print" option.

[0571] Step 12:

[0572] Terminal: Convert the .obj file into a format suitable for a 3D printer (e.g. G-code) and send it to the 3D printer.

[0573] Step 13:

[0574] Device: Once the print setup is complete, start your 3D print.

[0575] Step 14:

[0576] User: Once the 3D printer has finished printing, they retrieve the physical item and place or use it where they want.

[0577] Step 15:

[0578] On the device: At the same time, the emotion engine acquires emotion information through facial expression recognition and voice analysis, which is then analyzed together with the input information in the chat interface.

[0579] Step 16:

[0580] Server: Reflects the user's emotional information obtained by the emotion engine and adjusts the shape and design when generating the data file. For example, if the user is relaxed, a calm design is suggested.

[0581] Step 17:

[0582] Terminal: The adjusted .obj file is displayed to the user again for final confirmation.

[0583] Step 18:

[0584] User: Once final confirmation is complete, send it to the 3D printer for final printing.

[0585] Through these steps, the system of the present invention is implemented to enable users to easily create and output custom items that fit perfectly into specific locations or objects, and can reflect the user's emotional information in the process.

[0586] Example 2

[0587] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0588] Conventional 3D model generation systems often require users to have detailed technical knowledge and skills to create custom items that fit specific shapes and sizes. Furthermore, they lack the ability to customize designs based on the user's emotional state, leaving the user experience open to improvement.

[0589] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0590] In this invention, the server includes depth sensor means for scanning an object, processing means for processing data acquired from the depth sensor means to generate a three-dimensional model, an emotion engine for analyzing a user's facial expressions and voice to acquire emotion information, and output means for transmitting the generated data file to an output device and generating an object. This allows users to easily create custom items without special skills and enables customization of designs based on emotion information.

[0591] "Depth sensor"

[0592] is a device for measuring the distance and shape of an object, and primarily uses optical technology to obtain depth information.

[0593] "Processing means"

[0594] is a computing device or software that uses acquired data to generate a three-dimensional model.

[0595] "Input means"

[0596] is an interface through which the user inputs customization requests, and through which the user's instructions are transmitted to the system.

[0597] "Emotion Engine"

[0598] is a technology or device for acquiring emotional information by analyzing a user's facial expressions and voice.

[0599] "Generation means"

[0600] is a technology or device that analyzes requests obtained from an input means and emotional information obtained from an emotion engine, and generates an appropriate data file based on a three-dimensional model.

[0601] Output means

[0602] is a device for sending the generated data file to an output device to generate a physical object.

[0603] "Three-dimensional model"

[0604] is a digital representation created based on the shape and size of an object, and refers to three-dimensional structure data composed of depth information.

[0605] "Emotional information"

[0606] This is psychological state data that is analyzed by the emotion engine from the user's facial expressions and voice.

[0607] Data File

[0608] is a file format in which the three-dimensional model data is output after being appropriately processed by the generating means.

[0609] This invention provides a system that allows users to easily create items that perfectly fit into specific locations or objects. The system includes a depth sensor, processing means, input means, an emotion engine, generation means, and output means. These components work together to generate custom items based on the user's requests and emotion information.

[0610] First, the user scans the object using a depth sensor (for example, an optical depth measurement device built into a smartphone). The user holds the smartphone and moves around the object, acquiring depth information. This information is sent in real time to a processing means, which generates a 3D model. Specifically, the smartphone application analyzes the depth data and updates the 3D model every 0.1 seconds. This model is displayed on the application's main screen, where the user can rotate and zoom the model.

[0611] Next, the user uses a chat interface within the application on the device to input customization requests for the generated 3D model. For example, a specific request such as "I want a cup holder with a diameter of 10 cm and a height of 5 cm" can be input into the device. At the same time, the emotion engine analyzes the user's facial expressions and voice to obtain emotional information. This analysis is performed using a camera and microphone. If the user is smiling or speaking in a calm voice, the emotion engine determines that the user is relaxed.

[0612] The desire and emotion information acquired from the input means is sent to the generation means. The generation means analyzes this information and adjusts the 3D model data to an appropriate shape and size. For example, if the user is relaxed, adjustments are made to incorporate calm design elements. This ultimately generates an .obj file that reflects the user's desire and emotion. A Python-based shape generation algorithm is used for this adjustment.

[0613] The generated .obj file is displayed to the user through the terminal, and the user can confirm its contents. After confirming, the user sends the generated data file from the terminal to the output means, i.e., the 3D printer. The 3D printer generates a physical item based on the .obj file.

[0614] As a concrete example, let's say a user wants to create a pen holder that fits perfectly on their desk at home. In this case, the user uses the optical depth measurement device on their smartphone to scan the desired area of ​​the desk. Meanwhile, the emotion engine confirms that the user is relaxed. Next, they input in the application, "I want to create a pen holder that is 10 cm tall and 5 cm wide." The generator analyzes this and generates an .obj file of the pen holder that incorporates calming design elements, incorporating the emotion information. Finally, the user can print this file on a 3D printer and place the finished pen holder on their desk.

[0615] An example prompt is:

[0616] "I would like to make a pen holder that is 10cm tall and 5cm wide. Please keep the design calm."

[0617] In this way, the system of the present invention provides a means for users to easily create custom items they need in their daily lives, and can be used without any special skills. Furthermore, the introduction of an emotion engine enables customization according to the user's emotional state, providing a more satisfying user experience.

[0618] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0619] A detailed explanation of the system's program divided into processing steps

[0620] Step 1: Acquiring and sending depth information

[0621] concrete action

[0622] The user launches the smartphone app and switches to scan mode. The device uses the smartphone's built-in optical depth measurement device to scan the area around an object designated by the user and collect depth information.

[0623] input

[0624] The user selects the object they want to scan and moves around while holding the smartphone.

[0625] Data processing and calculation

[0626] The device processes the acquired depth data in real time and generates three-dimensional coordinate data.

[0627] output

[0628] The processed three-dimensional coordinate data is generated and sent to the next step.

[0629] Step 2: Generate a 3D model

[0630] concrete action

[0631] The device receives the processed depth data in real time and generates a 3D model, which the user can view on the application screen and manipulate, such as rotating and zooming.

[0632] input

[0633] Three-dimensional coordinate data obtained by scanning.

[0634] Data processing and calculation

[0635] The processing means builds a model based on the 3D coordinate data, and updates the model every 0.1 seconds.

[0636] output

[0637] The generated three-dimensional model is displayed in real time on the user's terminal screen.

[0638] Step 3: Enter your customization request

[0639] concrete action

[0640] The user uses the chat interface within the application on the device to input customization requests for the generated 3D model. For example, the user might input, "I want to create a cup holder that is 10 cm in diameter and 5 cm in height."

[0641] input

[0642] Specific customization requests entered by the user into the chat interface.

[0643] Data processing and calculation

[0644] The user's request is analyzed as text and passed to the generating means as an appropriate parameter.

[0645] output

[0646] The desired parameters after text analysis are sent to the next step.

[0647] Step 4: Acquiring emotional information

[0648] concrete action

[0649] Using a camera and microphone, the emotion engine analyzes the user's facial expressions and tone of voice. For example, if the user is smiling or speaking in a calm voice, it determines that the user is relaxed.

[0650] input

[0651] The user's facial expression data and voice data.

[0652] Data processing and calculation

[0653] The emotion engine uses facial recognition and voice analysis algorithms to determine the user's emotional state.

[0654] output

[0655] The analyzed emotion information is sent to the next step.

[0656] Step 5: Analyze and adjust the information

[0657] concrete action

[0658] The terminal receives the user's customization request and emotional information, and the generation means analyzes this information and adjusts the 3D model appropriately. For example, if the user is relaxed, an .obj file incorporating calm design elements is generated.

[0659] input

[0660] User customization request parameters and emotional information.

[0661] Data processing and calculation

[0662] The generating means adjusts the shape and size of the three-dimensional model based on the shape generation algorithm and generates an .obj file.

[0663] output

[0664] The generated .obj file is sent to the next step.

[0665] Step 6: Check and output the results

[0666] concrete action

[0667] The generated .obj file is displayed on the terminal screen for the user to review. Once the user confirms the design, the file is sent to the 3D printer, which creates the physical item based on the .obj file.

[0668] input

[0669] The generated .obj file.

[0670] Data processing and calculation

[0671] The system uses file transfer protocol to send the .obj file to the 3D printer.

[0672] output

[0673] The physical item is produced by a 3D printer.

[0674] In this way, each processing step has specific inputs and outputs, and specific actions are taken to make the entire system function.

[0675] (Application example 2)

[0676] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0677] Modern factory manufacturing processes require the rapid creation of specialized tools and parts. However, traditional manufacturing methods require significant time and effort to design and create specialized tools, significantly reducing the efficiency of factory workers. Furthermore, existing systems struggle to automatically generate custom tools that reflect user feedback and emotions. As a result, the flexibility and efficiency of the manufacturing process are limited. To address these challenges, a new system is needed that can generate custom tools based on user requests and emotion information in a fast and automated manner.

[0678] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a depth sensor means for scanning an object, a processing means for processing data acquired from the depth sensor means to generate a three-dimensional model, an input means for inputting a user's desires and emotional information for the generated three-dimensional model, an output means for transmitting the generated data file to an output device and generating an object, and an industrial robot means for generating an automatically adjusted custom tool based on the analyzed desires and emotional information. This enables factory workers to quickly and efficiently create specialized tools. Furthermore, providing custom tools that reflect the user's emotional information significantly improves the flexibility and efficiency of the manufacturing process.

[0679] The "depth sensor means" is a device for detecting the depth and distance information of an object.

[0680] The "processing means" refers to a device or program for generating a three-dimensional model based on the acquired depth data.

[0681] "Input means" refers to a device or interface that allows a user to input requests and emotional information to a three-dimensional model.

[0682] "Generation means" refers to a device or program that analyzes the user's requests and emotional information and generates a data file based on a three-dimensional model.

[0683] An "output means" is a device or interface for converting the generated data file into a physical form.

[0684] An "industrial robotic tool" is a device such as a robotic arm for automatically creating custom tools or objects based on a generated data file.

[0685] "User requirements" are specific requests that users have for the system, such as the shape, size, and functionality of custom tools or objects.

[0686] "Emotion information" is data that indicates the emotional state of the user, obtained from facial expressions, voice, etc.

[0687] A "3D model" is three-dimensional digital data that represents the depth and shape of a scanned object.

[0688] The "data file" is the design data of a custom tool based on a three-dimensional model or user requirements, which is created by the generating means.

[0689] "Custom Tools" are specialized tools and parts designed and produced based on a user's specific requirements.

[0690] To implement this invention, a system is constructed in the following steps: First, a user scans an object using a depth sensor, which can be a smartphone or tablet equipped with an optical depth measurement device, and data on the object's depth and shape is acquired.

[0691] This depth data is then sent in real time to a processing unit within the device, which generates a three-dimensional model using 3D modeling software such as Blender. Users can also use input units within the smartphone application to enter requests. For example, they can use a chat interface to enter a specific request, such as "I want a wrench with a diameter of 5 cm."

[0692] Furthermore, the user's emotional information is acquired through the device's camera and microphone and analyzed by an emotion engine (e.g., Microsoft Azure Face API), allowing customization according to the user's emotional state.

[0693] The generation means adjusts the shape and size of the 3D model data appropriately based on the user's requests and emotional information. A dedicated algorithm analyzes this, and finally generates a data file in .obj file format. This data file is sent by the server to a robot arm (e.g., Universal Robots' UR5). The server instructs the robot arm to create a custom tool based on the generated data file.

[0694] Finally, the robotic arm creates a custom tool based on the generated .obj file, which is then physically printed using a 3D printer (e.g., Formlabs Form 3) as the output method, allowing users to quickly obtain the specialized tool they desire.

[0695] As a concrete example, imagine a factory worker needs a specific tool that perfectly fits the design of a new product line. In this case, the worker uses their smartphone to scan an existing part, and the emotion engine detects their emotional state as relaxed. They then type, "I need a special wrench that's 10cm tall and 5cm wide" into the application's chat interface. The generator analyzes this, automatically adjusts the design of the custom tool based on the emotion information, and generates an .obj file. A robotic arm then creates the tool based on this design, and finally, a 3D printer physically prints it. The worker can then use the tool immediately.

[0696] Here is an example of a prompt to input to a generative AI model:

[0697] Based on the invention below, design an application for a custom tool-making system for factory workers.

[0698] Scan objects using depth sensing means

[0699] The acquired depth data is processed in real time to generate a 3D model

[0700] Enter user requests using the chat interface

[0701] Acquire and analyze user emotional information using an emotion engine

[0702] Generate .obj files based on 3D model data

[0703] The generated data file is sent to the robot arm

[0704] Robotic arm automatically generates custom tools

[0705] Finally, the custom tool is printed on a 3D printer.

[0706] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0707] Step 1:

[0708] The user scans the object using the depth sensor means. The depth sensor means uses an optical depth measurement device on a smartphone or tablet to acquire data on the depth and shape of the object. The input is the object to be scanned, and the output is the depth data of the object. The user holds the smartphone and moves around the object to be scanned.

[0709] Step 2:

[0710] The acquired depth data is sent in real time to the processing means within the device. As the processing means, 3D modeling software (e.g. Blender) is used to generate a three-dimensional model based on the depth data. The input is depth data and the output is a three-dimensional model. The device analyzes the depth data and generates a three-dimensional model of the object.

[0711] Step 3:

[0712] The user uses a chat interface within the smartphone application to input their requests for the generated 3D model. Text input or voice input can be used as the input method. The input is the user's request (e.g., "I want a wrench with a diameter of 5 cm"), and the output is the request information. The user inputs a specific request.

[0713] Step 4:

[0714] The device acquires the user's emotional information through the camera and microphone and analyzes it using an emotion engine (e.g., Microsoft Azure Face API). The input is the user's facial expression and voice, and the output is the analyzed emotional information. The device determines the user's emotional state.

[0715] Step 5:

[0716] The generator adjusts the shape and size of the 3D model data appropriately based on the user's request and emotional information. A dedicated algorithm analyzes this and finally generates a data file in .obj file format. The input is request information and emotional information, and the output is the adjusted 3D model data and .obj file. The generator automatically adjusts the design.

[0717] Step 6:

[0718] The server sends the generated data file to an industrial robotic means (e.g., a robot arm). The input is an .obj file, and the output is the data file transfer to the robot arm. The server sends instructions to the robot arm to process the generated model.

[0719] Step 7:

[0720] The robot arm creates a custom tool based on the submitted .obj file. The input is the .obj file, and the output is the physical custom tool. The robot arm then machines the tool into the specified shape.

[0721] Step 8:

[0722] Optionally, the custom tool is physically printed with a 3D printer as the output medium. The input is an .obj file and the output is a 3D printed custom tool. The 3D printer generates the physical tool.

[0723] The above process provides a system that allows users to quickly and efficiently create specialized tools.

[0724] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0725] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0726] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0727] [Third embodiment]

[0728] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0729] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0730] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0731] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0732] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0733] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0734] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0735] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0736] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0737] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0738] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0739] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0740] The present invention provides a system that allows users to easily create something that fits perfectly into a specific location or object. The system includes a depth sensor, a processing unit, an input unit, a generating unit, and an output unit, and these components work together to generate a custom item based on the user's request.

[0741] First, a user scans an object using a depth sensor (e.g., a LiDAR sensor built into a smartphone). The user holds the smartphone and moves it around the object to acquire depth information. This information is sent in real time to a processing unit, which then generates a 3D model.

[0742] Next, the user uses a chat interface within the application on the device to input customization requests for the generated 3D model. For example, the user can input a specific request such as "I want a cup holder with a diameter of 10 cm and a height of 5 cm." This input information is analyzed and sent to the generation means.

[0743] The generator analyzes the user's requests and adjusts the shape and size of the 3D model data appropriately using a dedicated algorithm, ultimately generating an .obj file that reflects the user's requests.

[0744] The generated .obj file is displayed to the user via the terminal, and the user can confirm its contents. After confirmation, the user sends the generated data file from the terminal to the output means, i.e., the 3D printer, which generates a physical item based on the .obj file.

[0745] As a concrete example, let's say a user wants to create a smartphone holder that fits perfectly inside a car's drink holder. In this case, the user uses the smartphone's LiDAR sensor to scan the inside of the drink holder. Next, in the application, the user enters, "I want to create a smartphone holder that is 7 cm in diameter and 10 cm high." The generator analyzes this and generates an .obj file of a smartphone holder shaped to fit inside the drink holder. Finally, the user can print this file on a 3D printer and use the finished smartphone holder in their car.

[0746] In this way, the system of the present invention provides a means for users to easily create custom items they need in their daily lives, and can be used without any special skills.

[0747] The processing flow will be explained below.

[0748] Step 1:

[0749] User: Launches the dedicated application on the smartphone and starts using the LiDAR sensor by pointing it at the object to be scanned. The user holds the smartphone and moves around the object to acquire depth information.

[0750] Step 2:

[0751] On the device: Depth data acquired by the LiDAR sensor is processed in real time to generate 3D point cloud data, which is then visually displayed in the application, allowing the user to follow the progress of the scan.

[0752] Step 3:

[0753] User: Once the scan is complete, press the "Scan Complete" button within the application to finalize the scan data.

[0754] Step 4:

[0755] Terminal: Compresses and converts the format of the confirmed point cloud data and sends it to a server via the Internet.

[0756] Step 5:

[0757] Server: Receives point cloud data sent from the device, verifies the data's normality and consistency, then generates a 3D mesh from the point cloud data and converts it into a format usable by the generation AI.

[0758] Step 6:

[0759] User: After completing the scan, the user accesses the chat interface within the application and enters their requirements and wishes for the item they want to create. For example, they could enter, "I want to make a smartphone holder with a diameter of 10 cm."

[0760] Step 7:

[0761] Terminal: The text analysis engine processes the user's input, extracts the appropriate instruction information, and sends it to the server.

[0762] Step 8:

[0763] Server: Based on instructions from the text analysis engine, the generation AI adjusts the shape and size of the 3D mesh data and generates an .obj file that meets the requirements.

[0764] Step 9:

[0765] Server: Sends the generated .obj files to the device.

[0766] Step 10:

[0767] Terminal: The received .obj file is displayed to the user, providing a preview, allowing the user to review the details of the result and request changes or corrections if necessary.

[0768] Step 11:

[0769] User: If there is no problem with the .obj file, select the "3D Print" option.

[0770] Step 12:

[0771] Terminal: Convert the .obj file into a format suitable for a 3D printer (e.g. G-code) and send it to the 3D printer.

[0772] Step 13:

[0773] Device: Once the print setup is complete, start your 3D print.

[0774] Step 14:

[0775] User: Once the 3D printer has finished printing, they retrieve the physical item and place or use it where they want.

[0776] These steps allow the system of the present invention to be implemented in a way that allows users to easily create and print custom items that fit perfectly into specific locations or objects.

[0777] Example 1

[0778] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0779] Previously, specialized knowledge and expensive equipment were required for users to create custom items that fit perfectly to specific objects or locations, making it difficult for ordinary users to create custom items they need in their daily lives.

[0780] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0781] In this invention, the server includes: means for scanning the periphery of an object using a depth sensor means and acquiring depth information; processing means for generating a 3D model in real time based on the depth information and transmitting it to the server; input means for providing a chat interface for users to input customization requests; generation means for analyzing the user's customization requests, adjusting the 3D model using a generative AI model, and generating an optimal data file; and output means for returning the generated data file to the terminal, displaying it for the user to confirm, and finally transmitting it to an output device. This enables general users to easily create custom items without special skills or expensive equipment.

[0782] A "depth sensor" is a device for measuring the distance and shape around an object.

[0783] A "three-dimensional model" is digital data that represents the shape and structure of an object or space in three-dimensional space.

[0784] A "server" is a computer system that processes and provides data in response to requests from multiple clients via a network.

[0785] "Processing means" refers to hardware or software for analyzing input data and performing necessary calculations or transformations.

[0786] A "generative AI model" is an artificial intelligence model used to generate and process data based on user instructions.

[0787] An "input means" is a device or software that includes an interface through which a user can input instructions or data into a system.

[0788] A "generator" is a mechanism for generating new data or models based on input data or instructions.

[0789] "Output means" refers to hardware or software for displaying and outputting the generated data and models.

[0790] A "prompt sentence" is a text sentence that a user enters to convey instructions or requests to the system.

[0791] A "3D viewer" is software for visually checking and manipulating 3D models.

[0792] A "3D printer" is a device used to create physical three-dimensional objects based on digital data.

[0793] The present invention provides a system that allows users to easily create custom items that fit perfectly into specific locations or objects. This system generates original items based on the user's requests by using a depth sensor, a processing unit, an input unit, a generating unit, and an output unit that work in cooperation with each other.

[0794] First, the user scans the object using the smartphone's built-in LiDAR sensor (depth sensor means). The user holds the smartphone and moves it around the object to acquire depth information. This depth information is sent in real time to the smartphone's processing means, and a 3D model is generated.

[0795] Next, the user inputs their customization requests for the generated 3D model through a chat interface (input means) provided within the smartphone application. For example, they might input a prompt such as, "I would like to create a smartphone holder with a diameter of 7 cm and a height of 10 cm." This input information is sent to the server (generation means), which analyzes the request using a generative AI model and generates a 3D model adjusted to the appropriate shape and size. The server then generates this 3D model as an .obj file and returns it to the device.

[0796] The user reviews the generated .obj file on their device. Using a 3D viewer, they can check the appearance and dimensions of the model. Once the user is satisfied with the content of the file, they send the .obj file from their device to a 3D printer (output means). The 3D printer generates a physical object based on the .obj file.

[0797] As a concrete example, consider a case where a user wants to create a smartphone holder that fits into a car's drink holder. In this case, the user uses the smartphone's LiDAR sensor to scan the inside of the drink holder. Then, in the application, the user enters, "I want to create a smartphone holder that is 7 cm in diameter and 10 cm high." The server's generator analyzes this request and generates an .obj file of a smartphone holder shaped to fit inside the drink holder. Finally, the user can print this file on a 3D printer and use the completed smartphone holder in their car.

[0798] Examples of prompts:

[0799] "I want to make a cup holder that is 10cm in diameter and 5cm in height."

[0800] "I want to make a seasoning rack that is 12cm wide, 15cm high, and 5cm deep."

[0801] "I want to create a custom case that fits perfectly to a specific shape."

[0802] In this way, the system of the present invention provides a means for users to easily create custom items needed in everyday life without having to have special skills.

[0803] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0804] Step 1:

[0805] The user scans an object using the smartphone's LiDAR sensor. The input is real-time depth information acquired while moving around the object. The depth information is sent to the device's processing means. The device generates a 3D model based on this and temporarily stores it in memory. The user can then check the status of the depth information acquisition in real time on the smartphone screen. The scan progress is displayed on the screen, along with the message "Scan complete."

[0806] Step 2:

[0807] The processing means of the terminal transmits the generated 3D model to the server. The generated 3D model is included as input. The 3D model is compressed and transmitted to the server via the network. The server receives the 3D model and begins analysis. During data transmission, a progress bar and messages are displayed.

[0808] Step 3:

[0809] The user uses the chat interface on the device to input their specific customization request. For example, they might input a prompt such as, "I want to create a smartphone holder that is 7 cm in diameter and 10 cm in height." The input contains the user's customization request. The device parses this prompt and sends it to the server. The input is then displayed in the chat interface, and a confirmation message is displayed.

[0810] Step 4:

[0811] The server uses a generative AI model to generate a custom model based on the received user customization requests and 3D model. The inputs are a prompt from the user and the 3D model. The server uses the generative AI model to execute an internal algorithm and generate a 3D model that matches the user's requests. This generated model is saved in .obj file format. Data processing is performed on the server side, and the progress is displayed.

[0812] Step 5:

[0813] The server sends the generated .obj file to the terminal. The input is the generated .obj file. The file is compressed and sent to the terminal over the network. The terminal receives the .obj file and notifies the user. As a result, the user's terminal displays "Custom model generated."

[0814] Step 6:

[0815] The user uses the terminal to check the generated .obj file. Using the 3D viewer, the appearance and dimensions of the model can be checked in detail. The input is the generated .obj file. The user checks and approves it. In operation, the 3D model is displayed on the screen and the user performs the check operation.

[0816] Step 7:

[0817] The user sends the .obj file from the device to the 3D printer, which is the output device. The input is the .obj file that the user has confirmed. The 3D printer generates the physical item based on this. As a result, the status of the 3D printer is displayed on the device screen, allowing the user to check the progress of the print. For example, the process of the user printing a smartphone holder is displayed in detail.

[0818] By following these steps, users can create custom items without any special expertise.

[0819] (Application example 1)

[0820] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0821] In modern factories, creating custom tools and parts to fit specific machine components is a complex and time-consuming task. Traditional methods require specialized knowledge and make it difficult to efficiently design and manufacture custom parts. This can lead to process interruptions and delays, reducing production efficiency, especially when rapid changes or modifications to production lines are required. To solve this problem, a system is needed that allows users to easily design and quickly manufacture custom parts.

[0822] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0823] In this invention, the server includes a sensor means for scanning an object, an information processing means for processing data acquired from the sensor means to generate a three-dimensional model, an input means for inputting a user's request for the generated three-dimensional model, a generation means for analyzing the request acquired from the input means and generating an appropriate data file based on the three-dimensional model, an output means for transmitting the generated data file to an output device to generate an object, and a control means implemented in a computer for controlling a machine tool appropriate for the generated data file, thereby enabling the rapid and efficient design and manufacturing of required custom tools and parts in a factory.

[0824] "Sensor means" refers to a device used to acquire information about the three-dimensional shape and depth of an object. Optical depth measurement devices and LiDAR sensors are mainly used.

[0825] "Information processing means" refers to a computer system that processes data acquired from the sensor means and generates a three-dimensional model. It analyzes, converts, and generates the data.

[0826] "Input means" refers to the interface through which the user inputs specific requests to the system. Chat interfaces and touch screens are typically used.

[0827] "Generation means" refers to a device or function that analyzes the 3D model and generates an appropriate data file based on the user's request input via the input means. The shape and size of the model are adjusted using an algorithm.

[0828] The "output means" is a device that generates the generated data file as a physical object. 3D printers and machine tools are usually used.

[0829] The "control means" is a system for controlling the machine tool based on the generated data file, and instructs the operation of the machine tool using a computer program.

[0830] Hardware Configuration

[0831] The system includes the following major hardware components:

[0832] 1. Sensor means: Optical depth measurement devices (such as LiDAR sensors) for obtaining three-dimensional shape and depth information of objects.

[0833] 2. Information processing means: A computer that processes the data obtained from the sensor means and generates a three-dimensional model.

[0834] 3. Input means: The interface through which users input their requests (e.g., a chat interface or touch screen on a smartphone).

[0835] 4. Generation means: A processing device that analyzes the three-dimensional model and generates a data file based on the user's request.

[0836] 5. Output means: A 3D printer or machine tool that produces a physical item based on the generated data file.

[0837] 6. Control means: A computer program that controls the machine tool based on the generated data file.

[0838] Software Structure and Algorithms

[0839] The system uses libraries such as open3d to process 3D point cloud data and generate .obj files, and also uses an AI model to analyze user input and generate an appropriate 3D model.

[0840] Operation procedures and data processing

[0841] The server processes the data and generates the custom items as follows:

[0842] 1. Data acquisition: Using sensor means to acquire depth information of the scanned object. For example, scanning a specific part in a factory with a smartphone's LiDAR sensor.

[0843] 2. 3D model generation: The acquired data is sent to the information processing means, and a 3D model is generated using the open3d library.

[0844] 3. Request input: The user uses the chat interface on their smartphone to input a specific request, for example, "I want to create an adapter that fits a pipe with a diameter of 5 cm and a length of 10 cm."

[0845] 4. Request analysis: The generator analyzes the input request and adjusts the 3D model. The shape and size are automatically adjusted using an AI model.

[0846] 5. Data file generation: The final adjusted 3D model is generated as a data file (.obj format).

[0847] 6. Output: The generated data file is sent to a 3D printer or machine tool to create the physical custom item.

[0848] 7. Machine Tool Control: A control means controls the machine tool based on the generated data file to rapidly manufacture precision custom parts.

[0849] Specific examples

[0850] For example, to create an adapter to fit a specific machine part in a factory, a user might do the following:

[0851] 1. Scan the target pipe with your smartphone's LiDAR sensor.

[0852] 2. Type into the chat interface, "I want to generate an adapter that fits a pipe that is 5cm in diameter and 10cm long."

[0853] 3. The server generates a 3D model based on the acquired data and adjusts it to suit your needs.

[0854] 4. The generated .obj file is sent to a 3D printer to produce the physical adapter.

[0855] Prompt Sentence Examples

[0856] "Create an .obj file that generates a custom adapter that fits a pipe that is 5cm in diameter and 10cm long."

[0857] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0858] Step 1:

[0859] A sensor means is used to obtain depth information of the scanned object.

[0860] Input: Object to scan

[0861] Data processing: A depth sensor (LiDAR sensor) scans the area around the object and obtains three-dimensional point cloud data.

[0862] Output: Acquired 3D point cloud data

[0863] Step 2:

[0864] The information processing means processes the acquired three-dimensional point cloud data to generate a three-dimensional model.

[0865] Input: 3D point cloud data

[0866] Data processing: Analyze the 3D point cloud data using the open3d library and generate a 3D model.

[0867] Output: Generated 3D model

[0868] Step 3:

[0869] The user uses the chat interface of the terminal to input a custom request.

[0870] Input: User's custom request (e.g. "I want to create an adapter that fits a pipe with a diameter of 5 cm and a length of 10 cm")

[0871] How it works: A user types a specific request into the chat interface on their smartphone or device.

[0872] Output: Input custom request

[0873] Step 4:

[0874] The generating means analyzes the input request and adjusts the three-dimensional model.

[0875] Input: Generated 3D model, custom request

[0876] Data calculation: Using a generative AI model, the shape and size of the 3D model are adjusted based on the input customization requests. Specifically, the system analyzes the user's requests and appropriately changes the shape parameters of the 3D model.

[0877] Output: Adjusted 3D model

[0878] Step 5:

[0879] Generate a suitable data file (.obj format) based on the adjusted 3D model.

[0880] Input: Calibrated 3D model

[0881] Data Generation: Export the 3D model in .obj file format.

[0882] Output: Generated .obj file

[0883] Step 6:

[0884] An output method is used to send the generated .obj file to a 3D printer or machine tool to manufacture the physical custom item.

[0885] Input: Generated .obj file

[0886] Data Transfer: Send the generated .obj file over the network to the 3D printer or machine tool.

[0887] How it works: The 3D printer deposits filament based on the .obj file to create a physical custom part.

[0888] Output: The generated physical custom item

[0889] Step 7:

[0890] The control means controls the machine tool based on the generated data file to manufacture precision custom parts.

[0891] Input: Generated .obj file

[0892] Control operation: The control program installed in the machine tool drives the machine tool based on the contents of the .obj file to manufacture a custom part with the specified shape.

[0893] Output: Precision-manufactured custom parts

[0894] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0895] The present invention provides a system that allows users to easily create items that perfectly fit into specific locations or objects. The system includes a depth sensor, a processing unit, an input unit, a generating unit, an output unit, and an emotion engine. These components work together to generate custom items based on the user's requests and emotion information.

[0896] First, a user scans an object using a depth sensor (e.g., an optical depth measurement device built into a smartphone). The user holds the smartphone and moves around the object to acquire depth information. This information is sent in real time to a processing unit, which then generates a three-dimensional model.

[0897] Next, the user uses a chat interface within the application on the device to input customization requests for the generated 3D model. For example, a specific request such as "I want a cup holder with a diameter of 10 cm and a height of 5 cm" can be input into the device. At the same time, the emotion engine analyzes the user's facial expressions and voice to obtain emotional information. This input information and emotional information are analyzed and sent to the generation means.

[0898] The generator analyzes the user's requests and emotional information and adjusts the shape and size of the 3D model data appropriately. For example, if the user is excited, bold elements are added to the design, and if the user is relaxed, the design is adjusted to be calm. A dedicated algorithm is used for this adjustment, and finally an .obj file that reflects the user's requests and emotions is generated.

[0899] The generated .obj file is displayed to the user via the terminal, and the user can confirm its contents. After confirmation, the user sends the generated data file from the terminal to the output means, i.e., the 3D printer, which generates a physical item based on the .obj file.

[0900] As a concrete example, let's say a user wants to create a pen holder that fits perfectly on their desk at home. In this case, the user uses the optical depth measurement device on their smartphone to scan the desired area of ​​the desk. Meanwhile, the emotion engine ensures that the user is relaxed. Next, they input in the application, "I want to create a pen holder that is 10cm tall and 5cm wide." The generator analyzes this and generates an .obj file of the pen holder that incorporates emotional information and calm design elements. Finally, the user can print this file on a 3D printer and place the finished pen holder on their desk.

[0901] In this way, the system of the present invention provides a means for users to easily create custom items they need in their daily lives, and can be used without any special skills. Furthermore, the introduction of an emotion engine enables customization according to the user's emotional state, providing a more satisfying user experience.

[0902] The processing flow will be explained below.

[0903] Step 1:

[0904] User: Launches the dedicated application on the smartphone and starts using the LiDAR sensor by pointing it at the object to be scanned. The user holds the smartphone and moves around the object to acquire depth information.

[0905] Step 2:

[0906] On the device: Depth data acquired by the LiDAR sensor is processed in real time to generate 3D point cloud data, which is then visually displayed in the application, allowing the user to follow the progress of the scan.

[0907] Step 3:

[0908] User: Once the scan is complete, press the "Scan Complete" button within the application to finalize the scan data.

[0909] Step 4:

[0910] Terminal: Compresses and converts the format of the confirmed point cloud data and sends it to a server via the Internet.

[0911] Step 5:

[0912] Server: Receives point cloud data sent from the device, verifies the data's normality and consistency, then generates a 3D mesh from the point cloud data and converts it into a format usable by the generation AI.

[0913] Step 6:

[0914] User: After completing the scan, the user accesses the chat interface within the application and enters their requirements and wishes for the item they wish to create, for example, "I would like to create a vase that is 10cm tall and 7cm in diameter."

[0915] Step 7:

[0916] Terminal: The text analysis engine processes the user's input, extracts the appropriate instruction information, and sends it to the server.

[0917] Step 8:

[0918] Server: Based on instructions from the text analysis engine, the generation AI adjusts the shape and size of the 3D mesh data and generates an .obj file that meets the requirements.

[0919] Step 9:

[0920] Server: Sends the generated .obj files to the device.

[0921] Step 10:

[0922] Terminal: The received .obj file is displayed to the user, providing a preview, allowing the user to review the details of the result and request changes or corrections if necessary.

[0923] Step 11:

[0924] User: If there is no problem with the .obj file, select the "3D Print" option.

[0925] Step 12:

[0926] Terminal: Convert the .obj file into a format suitable for a 3D printer (e.g. G-code) and send it to the 3D printer.

[0927] Step 13:

[0928] Device: Once the print setup is complete, start your 3D print.

[0929] Step 14:

[0930] User: Once the 3D printer has finished printing, they retrieve the physical item and place or use it where they want.

[0931] Step 15:

[0932] On the device: At the same time, the emotion engine acquires emotion information through facial expression recognition and voice analysis, which is then analyzed together with the input information in the chat interface.

[0933] Step 16:

[0934] Server: Reflects the user's emotional information obtained by the emotion engine and adjusts the shape and design when generating the data file. For example, if the user is relaxed, a calm design is suggested.

[0935] Step 17:

[0936] Terminal: The adjusted .obj file is displayed to the user again for final confirmation.

[0937] Step 18:

[0938] User: Once final confirmation is complete, send it to the 3D printer for final printing.

[0939] Through these steps, the system of the present invention is implemented to enable users to easily create and output custom items that fit perfectly into specific locations or objects, and can reflect the user's emotional information in the process.

[0940] Example 2

[0941] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0942] Conventional 3D model generation systems often require users to have detailed technical knowledge and skills to create custom items that fit specific shapes and sizes. Furthermore, they lack the ability to customize designs based on the user's emotional state, leaving the user experience open to improvement.

[0943] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0944] In this invention, the server includes depth sensor means for scanning an object, processing means for processing data acquired from the depth sensor means to generate a three-dimensional model, an emotion engine for analyzing a user's facial expressions and voice to acquire emotion information, and output means for transmitting the generated data file to an output device and generating an object. This allows users to easily create custom items without special skills and enables customization of designs based on emotion information.

[0945] "Depth sensor"

[0946] is a device for measuring the distance and shape of an object, and primarily uses optical technology to obtain depth information.

[0947] "Processing means"

[0948] is a computing device or software that uses acquired data to generate a three-dimensional model.

[0949] "Input means"

[0950] is an interface through which the user inputs customization requests, and through which the user's instructions are transmitted to the system.

[0951] "Emotion Engine"

[0952] is a technology or device for acquiring emotional information by analyzing a user's facial expressions and voice.

[0953] "Generation means"

[0954] is a technology or device that analyzes requests obtained from an input means and emotional information obtained from an emotion engine, and generates an appropriate data file based on a three-dimensional model.

[0955] Output means

[0956] is a device for sending the generated data file to an output device to generate a physical object.

[0957] "Three-dimensional model"

[0958] is a digital representation created based on the shape and size of an object, and refers to three-dimensional structure data composed of depth information.

[0959] "Emotional information"

[0960] This is psychological state data that is analyzed by the emotion engine from the user's facial expressions and voice.

[0961] Data File

[0962] is a file format in which the three-dimensional model data is output after being appropriately processed by the generating means.

[0963] This invention provides a system that allows users to easily create items that perfectly fit into specific locations or objects. The system includes a depth sensor, processing means, input means, an emotion engine, generation means, and output means. These components work together to generate custom items based on the user's requests and emotion information.

[0964] First, the user scans the object using a depth sensor (for example, an optical depth measurement device built into a smartphone). The user holds the smartphone and moves around the object, acquiring depth information. This information is sent in real time to a processing means, which generates a 3D model. Specifically, the smartphone application analyzes the depth data and updates the 3D model every 0.1 seconds. This model is displayed on the application's main screen, where the user can rotate and zoom the model.

[0965] Next, the user uses a chat interface within the application on the device to input customization requests for the generated 3D model. For example, a specific request such as "I want a cup holder with a diameter of 10 cm and a height of 5 cm" can be input into the device. At the same time, the emotion engine analyzes the user's facial expressions and voice to obtain emotional information. This analysis is performed using a camera and microphone. If the user is smiling or speaking in a calm voice, the emotion engine determines that the user is relaxed.

[0966] The desire and emotion information acquired from the input means is sent to the generation means. The generation means analyzes this information and adjusts the 3D model data to an appropriate shape and size. For example, if the user is relaxed, adjustments are made to incorporate calm design elements. This ultimately generates an .obj file that reflects the user's desire and emotion. A Python-based shape generation algorithm is used for this adjustment.

[0967] The generated .obj file is displayed to the user through the terminal, and the user can confirm its contents. After confirming, the user sends the generated data file from the terminal to the output means, i.e., the 3D printer. The 3D printer generates a physical item based on the .obj file.

[0968] As a concrete example, let's say a user wants to create a pen holder that fits perfectly on their desk at home. In this case, the user uses the optical depth measurement device on their smartphone to scan the desired area of ​​the desk. Meanwhile, the emotion engine confirms that the user is relaxed. Next, they input in the application, "I want to create a pen holder that is 10 cm tall and 5 cm wide." The generator analyzes this and generates an .obj file of the pen holder that incorporates calming design elements, incorporating the emotion information. Finally, the user can print this file on a 3D printer and place the finished pen holder on their desk.

[0969] An example prompt is:

[0970] "I would like to make a pen holder that is 10cm tall and 5cm wide. Please keep the design calm."

[0971] In this way, the system of the present invention provides a means for users to easily create custom items they need in their daily lives, and can be used without any special skills. Furthermore, the introduction of an emotion engine enables customization according to the user's emotional state, providing a more satisfying user experience.

[0972] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0973] A detailed explanation of the system's program divided into processing steps

[0974] Step 1: Acquiring and sending depth information

[0975] concrete action

[0976] The user launches the smartphone app and switches to scan mode. The device uses the smartphone's built-in optical depth measurement device to scan the area around an object designated by the user and collect depth information.

[0977] input

[0978] The user selects the object they want to scan and moves around while holding the smartphone.

[0979] Data processing and calculation

[0980] The device processes the acquired depth data in real time and generates three-dimensional coordinate data.

[0981] output

[0982] The processed three-dimensional coordinate data is generated and sent to the next step.

[0983] Step 2: Generate a 3D model

[0984] concrete action

[0985] The device receives the processed depth data in real time and generates a 3D model, which the user can view on the application screen and manipulate, such as rotating and zooming.

[0986] input

[0987] Three-dimensional coordinate data obtained by scanning.

[0988] Data processing and calculation

[0989] The processing means builds a model based on the 3D coordinate data, and updates the model every 0.1 seconds.

[0990] output

[0991] The generated three-dimensional model is displayed in real time on the user's terminal screen.

[0992] Step 3: Enter your customization request

[0993] concrete action

[0994] The user uses the chat interface within the application on the device to input customization requests for the generated 3D model. For example, the user might input, "I want to create a cup holder that is 10 cm in diameter and 5 cm in height."

[0995] input

[0996] Specific customization requests entered by the user into the chat interface.

[0997] Data processing and calculation

[0998] The user's request is analyzed as text and passed to the generating means as an appropriate parameter.

[0999] output

[1000] The desired parameters after text analysis are sent to the next step.

[1001] Step 4: Acquiring emotional information

[1002] concrete action

[1003] Using a camera and microphone, the emotion engine analyzes the user's facial expressions and tone of voice. For example, if the user is smiling or speaking in a calm voice, it determines that the user is relaxed.

[1004] input

[1005] The user's facial expression data and voice data.

[1006] Data processing and calculation

[1007] The emotion engine uses facial recognition and voice analysis algorithms to determine the user's emotional state.

[1008] output

[1009] The analyzed emotion information is sent to the next step.

[1010] Step 5: Analyze and adjust the information

[1011] concrete action

[1012] The terminal receives the user's customization request and emotional information, and the generation means analyzes this information and adjusts the 3D model appropriately. For example, if the user is relaxed, an .obj file incorporating calm design elements is generated.

[1013] input

[1014] User customization request parameters and emotional information.

[1015] Data processing and calculation

[1016] The generating means adjusts the shape and size of the three-dimensional model based on the shape generation algorithm and generates an .obj file.

[1017] output

[1018] The generated .obj file is sent to the next step.

[1019] Step 6: Check and output the results

[1020] concrete action

[1021] The generated .obj file is displayed on the terminal screen for the user to review. Once the user confirms the design, the file is sent to the 3D printer, which creates the physical item based on the .obj file.

[1022] input

[1023] The generated .obj file.

[1024] Data processing and calculation

[1025] The system uses file transfer protocol to send the .obj file to the 3D printer.

[1026] output

[1027] The physical item is produced by a 3D printer.

[1028] In this way, each processing step has specific inputs and outputs, and specific actions are taken to make the entire system function.

[1029] (Application example 2)

[1030] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1031] Modern factory manufacturing processes require the rapid creation of specialized tools and parts. However, traditional manufacturing methods require significant time and effort to design and create specialized tools, significantly reducing the efficiency of factory workers. Furthermore, existing systems struggle to automatically generate custom tools that reflect user feedback and emotions. As a result, the flexibility and efficiency of the manufacturing process are limited. To address these challenges, a new system is needed that can generate custom tools based on user requests and emotion information in a fast and automated manner.

[1032] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a depth sensor means for scanning an object, a processing means for processing data acquired from the depth sensor means to generate a three-dimensional model, an input means for inputting a user's desires and emotional information for the generated three-dimensional model, an output means for transmitting the generated data file to an output device and generating an object, and an industrial robot means for generating an automatically adjusted custom tool based on the analyzed desires and emotional information. This enables factory workers to quickly and efficiently create specialized tools. Furthermore, providing custom tools that reflect the user's emotional information significantly improves the flexibility and efficiency of the manufacturing process.

[1033] The "depth sensor means" is a device for detecting the depth and distance information of an object.

[1034] The "processing means" refers to a device or program for generating a three-dimensional model based on the acquired depth data.

[1035] "Input means" refers to a device or interface that allows a user to input requests and emotional information to a three-dimensional model.

[1036] "Generation means" refers to a device or program that analyzes the user's requests and emotional information and generates a data file based on a three-dimensional model.

[1037] An "output means" is a device or interface for converting the generated data file into a physical form.

[1038] An "industrial robotic tool" is a device such as a robotic arm for automatically creating custom tools or objects based on a generated data file.

[1039] "User requirements" are specific requests that users have for the system, such as the shape, size, and functionality of custom tools or objects.

[1040] "Emotion information" is data that indicates the emotional state of the user, obtained from facial expressions, voice, etc.

[1041] A "3D model" is three-dimensional digital data that represents the depth and shape of a scanned object.

[1042] The "data file" is the design data of a custom tool based on a three-dimensional model or user requirements, which is created by the generating means.

[1043] "Custom Tools" are specialized tools and parts designed and produced based on a user's specific requirements.

[1044] To implement this invention, a system is constructed in the following steps: First, a user scans an object using a depth sensor, which can be a smartphone or tablet equipped with an optical depth measurement device, and data on the object's depth and shape is acquired.

[1045] This depth data is then sent in real time to a processing unit within the device, which generates a three-dimensional model using 3D modeling software such as Blender. Users can also use input units within the smartphone application to enter requests. For example, they can use a chat interface to enter a specific request, such as "I want a wrench with a diameter of 5 cm."

[1046] Furthermore, the user's emotional information is acquired through the device's camera and microphone and analyzed by an emotion engine (e.g., Microsoft Azure Face API), allowing customization according to the user's emotional state.

[1047] The generation means adjusts the shape and size of the 3D model data appropriately based on the user's requests and emotional information. A dedicated algorithm analyzes this, and finally generates a data file in .obj file format. This data file is sent by the server to a robot arm (e.g., Universal Robots' UR5). The server instructs the robot arm to create a custom tool based on the generated data file.

[1048] Finally, the robotic arm creates a custom tool based on the generated .obj file, which is then physically printed using a 3D printer (e.g., Formlabs Form 3) as the output method, allowing users to quickly obtain the specialized tool they desire.

[1049] As a concrete example, imagine a factory worker needs a specific tool that perfectly fits the design of a new product line. In this case, the worker uses their smartphone to scan an existing part, and the emotion engine detects their emotional state as relaxed. They then type, "I need a special wrench that's 10cm tall and 5cm wide" into the application's chat interface. The generator analyzes this, automatically adjusts the design of the custom tool based on the emotion information, and generates an .obj file. A robotic arm then creates the tool based on this design, and finally, a 3D printer physically prints it. The worker can then use the tool immediately.

[1050] Here is an example of a prompt to input to a generative AI model:

[1051] Based on the invention below, design an application for a custom tool-making system for factory workers.

[1052] Scan objects using depth sensing means

[1053] The acquired depth data is processed in real time to generate a 3D model

[1054] Enter user requests using the chat interface

[1055] Acquire and analyze user emotional information using an emotion engine

[1056] Generate .obj files based on 3D model data

[1057] The generated data file is sent to the robot arm

[1058] Robotic arm automatically generates custom tools

[1059] Finally, the custom tool is printed on a 3D printer.

[1060] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1061] Step 1:

[1062] The user scans the object using the depth sensor means. The depth sensor means uses an optical depth measurement device on a smartphone or tablet to acquire data on the depth and shape of the object. The input is the object to be scanned, and the output is the depth data of the object. The user holds the smartphone and moves around the object to be scanned.

[1063] Step 2:

[1064] The acquired depth data is sent in real time to the processing means within the device. As the processing means, 3D modeling software (e.g. Blender) is used to generate a three-dimensional model based on the depth data. The input is depth data and the output is a three-dimensional model. The device analyzes the depth data and generates a three-dimensional model of the object.

[1065] Step 3:

[1066] The user uses a chat interface within the smartphone application to input their requests for the generated 3D model. Text input or voice input can be used as the input method. The input is the user's request (e.g., "I want a wrench with a diameter of 5 cm"), and the output is the request information. The user inputs a specific request.

[1067] Step 4:

[1068] The device acquires the user's emotional information through the camera and microphone and analyzes it using an emotion engine (e.g., Microsoft Azure Face API). The input is the user's facial expression and voice, and the output is the analyzed emotional information. The device determines the user's emotional state.

[1069] Step 5:

[1070] The generator adjusts the shape and size of the 3D model data appropriately based on the user's request and emotional information. A dedicated algorithm analyzes this and finally generates a data file in .obj file format. The input is request information and emotional information, and the output is the adjusted 3D model data and .obj file. The generator automatically adjusts the design.

[1071] Step 6:

[1072] The server sends the generated data file to an industrial robotic means (e.g., a robot arm). The input is an .obj file, and the output is the data file transfer to the robot arm. The server sends instructions to the robot arm to process the generated model.

[1073] Step 7:

[1074] The robot arm creates a custom tool based on the submitted .obj file. The input is the .obj file, and the output is the physical custom tool. The robot arm then machines the tool into the specified shape.

[1075] Step 8:

[1076] Optionally, the custom tool is physically printed with a 3D printer as the output medium. The input is an .obj file and the output is a 3D printed custom tool. The 3D printer generates the physical tool.

[1077] The above process provides a system that allows users to quickly and efficiently create specialized tools.

[1078] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1079] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1080] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1081] [Fourth embodiment]

[1082] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1083] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1084] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1085] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1086] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1087] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1088] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1089] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1090] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1091] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1092] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1093] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1094] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1095] The present invention provides a system that allows users to easily create something that fits perfectly into a specific location or object. The system includes a depth sensor, a processing unit, an input unit, a generating unit, and an output unit, and these components work together to generate a custom item based on the user's request.

[1096] First, a user scans an object using a depth sensor (e.g., a LiDAR sensor built into a smartphone). The user holds the smartphone and moves it around the object to acquire depth information. This information is sent in real time to a processing unit, which then generates a 3D model.

[1097] Next, the user uses a chat interface within the application on the device to input customization requests for the generated 3D model. For example, the user can input a specific request such as "I want a cup holder with a diameter of 10 cm and a height of 5 cm." This input information is analyzed and sent to the generation means.

[1098] The generator analyzes the user's requests and adjusts the shape and size of the 3D model data appropriately using a dedicated algorithm, ultimately generating an .obj file that reflects the user's requests.

[1099] The generated .obj file is displayed to the user via the terminal, and the user can confirm its contents. After confirmation, the user sends the generated data file from the terminal to the output means, i.e., the 3D printer, which generates a physical item based on the .obj file.

[1100] As a concrete example, let's say a user wants to create a smartphone holder that fits perfectly inside a car's drink holder. In this case, the user uses the smartphone's LiDAR sensor to scan the inside of the drink holder. Next, in the application, the user enters, "I want to create a smartphone holder that is 7 cm in diameter and 10 cm high." The generator analyzes this and generates an .obj file of a smartphone holder shaped to fit inside the drink holder. Finally, the user can print this file on a 3D printer and use the finished smartphone holder in their car.

[1101] In this way, the system of the present invention provides a means for users to easily create custom items they need in their daily lives, and can be used without any special skills.

[1102] The processing flow will be explained below.

[1103] Step 1:

[1104] User: Launches the dedicated application on the smartphone and starts using the LiDAR sensor by pointing it at the object to be scanned. The user holds the smartphone and moves around the object to acquire depth information.

[1105] Step 2:

[1106] On the device: Depth data acquired by the LiDAR sensor is processed in real time to generate 3D point cloud data, which is then visually displayed in the application, allowing the user to follow the progress of the scan.

[1107] Step 3:

[1108] User: Once the scan is complete, press the "Scan Complete" button within the application to finalize the scan data.

[1109] Step 4:

[1110] Terminal: Compresses and converts the format of the confirmed point cloud data and sends it to a server via the Internet.

[1111] Step 5:

[1112] Server: Receives point cloud data sent from the device, verifies the data's normality and consistency, then generates a 3D mesh from the point cloud data and converts it into a format usable by the generation AI.

[1113] Step 6:

[1114] User: After completing the scan, the user accesses the chat interface within the application and enters their requirements and wishes for the item they want to create. For example, they could enter, "I want to make a smartphone holder with a diameter of 10 cm."

[1115] Step 7:

[1116] Terminal: The text analysis engine processes the user's input, extracts the appropriate instruction information, and sends it to the server.

[1117] Step 8:

[1118] Server: Based on instructions from the text analysis engine, the generation AI adjusts the shape and size of the 3D mesh data and generates an .obj file that meets the requirements.

[1119] Step 9:

[1120] Server: Sends the generated .obj files to the device.

[1121] Step 10:

[1122] Terminal: The received .obj file is displayed to the user, providing a preview, allowing the user to review the details of the result and request changes or corrections if necessary.

[1123] Step 11:

[1124] User: If there is no problem with the .obj file, select the "3D Print" option.

[1125] Step 12:

[1126] Terminal: Convert the .obj file into a format suitable for a 3D printer (e.g. G-code) and send it to the 3D printer.

[1127] Step 13:

[1128] Device: Once the print setup is complete, start your 3D print.

[1129] Step 14:

[1130] User: Once the 3D printer has finished printing, they retrieve the physical item and place or use it where they want.

[1131] These steps allow the system of the present invention to be implemented in a way that allows users to easily create and print custom items that fit perfectly into specific locations or objects.

[1132] Example 1

[1133] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1134] Previously, specialized knowledge and expensive equipment were required for users to create custom items that fit perfectly to specific objects or locations, making it difficult for ordinary users to create custom items they need in their daily lives.

[1135] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1136] In this invention, the server includes: means for scanning the periphery of an object using a depth sensor means and acquiring depth information; processing means for generating a 3D model in real time based on the depth information and transmitting it to the server; input means for providing a chat interface for users to input customization requests; generation means for analyzing the user's customization requests, adjusting the 3D model using a generative AI model, and generating an optimal data file; and output means for returning the generated data file to the terminal, displaying it for the user to confirm, and finally transmitting it to an output device. This enables general users to easily create custom items without special skills or expensive equipment.

[1137] A "depth sensor" is a device for measuring the distance and shape around an object.

[1138] A "three-dimensional model" is digital data that represents the shape and structure of an object or space in three-dimensional space.

[1139] A "server" is a computer system that processes and provides data in response to requests from multiple clients via a network.

[1140] "Processing means" refers to hardware or software for analyzing input data and performing necessary calculations or transformations.

[1141] A "generative AI model" is an artificial intelligence model used to generate and process data based on user instructions.

[1142] An "input means" is a device or software that includes an interface through which a user can input instructions or data into a system.

[1143] A "generator" is a mechanism for generating new data or models based on input data or instructions.

[1144] "Output means" refers to hardware or software for displaying and outputting the generated data and models.

[1145] A "prompt sentence" is a text sentence that a user enters to convey instructions or requests to the system.

[1146] A "3D viewer" is software for visually checking and manipulating 3D models.

[1147] A "3D printer" is a device used to create physical three-dimensional objects based on digital data.

[1148] The present invention provides a system that allows users to easily create custom items that fit perfectly into specific locations or objects. This system generates original items based on the user's requests by using a depth sensor, a processing unit, an input unit, a generating unit, and an output unit that work in cooperation with each other.

[1149] First, the user scans the object using the smartphone's built-in LiDAR sensor (depth sensor means). The user holds the smartphone and moves it around the object to acquire depth information. This depth information is sent in real time to the smartphone's processing means, and a 3D model is generated.

[1150] Next, the user inputs their customization requests for the generated 3D model through a chat interface (input means) provided within the smartphone application. For example, they might input a prompt such as, "I would like to create a smartphone holder with a diameter of 7 cm and a height of 10 cm." This input information is sent to the server (generation means), which analyzes the request using a generative AI model and generates a 3D model adjusted to the appropriate shape and size. The server then generates this 3D model as an .obj file and returns it to the device.

[1151] The user reviews the generated .obj file on their device. Using a 3D viewer, they can check the appearance and dimensions of the model. Once the user is satisfied with the content of the file, they send the .obj file from their device to a 3D printer (output means). The 3D printer generates a physical object based on the .obj file.

[1152] As a concrete example, consider a case where a user wants to create a smartphone holder that fits into a car's drink holder. In this case, the user uses the smartphone's LiDAR sensor to scan the inside of the drink holder. Then, in the application, the user enters, "I want to create a smartphone holder that is 7 cm in diameter and 10 cm high." The server's generator analyzes this request and generates an .obj file of a smartphone holder shaped to fit inside the drink holder. Finally, the user can print this file on a 3D printer and use the completed smartphone holder in their car.

[1153] Examples of prompts:

[1154] "I want to make a cup holder that is 10cm in diameter and 5cm in height."

[1155] "I want to make a seasoning rack that is 12cm wide, 15cm high, and 5cm deep."

[1156] "I want to create a custom case that fits perfectly to a specific shape."

[1157] In this way, the system of the present invention provides a means for users to easily create custom items needed in everyday life without having to have special skills.

[1158] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1159] Step 1:

[1160] The user scans an object using the smartphone's LiDAR sensor. The input is real-time depth information acquired while moving around the object. The depth information is sent to the device's processing means. The device generates a 3D model based on this and temporarily stores it in memory. The user can then check the status of the depth information acquisition in real time on the smartphone screen. The scan progress is displayed on the screen, along with the message "Scan complete."

[1161] Step 2:

[1162] The processing means of the terminal transmits the generated 3D model to the server. The generated 3D model is included as input. The 3D model is compressed and transmitted to the server via the network. The server receives the 3D model and begins analysis. During data transmission, a progress bar and messages are displayed.

[1163] Step 3:

[1164] The user uses the chat interface on the device to input their specific customization request. For example, they might input a prompt such as, "I want to create a smartphone holder that is 7 cm in diameter and 10 cm in height." The input contains the user's customization request. The device parses this prompt and sends it to the server. The input is then displayed in the chat interface, and a confirmation message is displayed.

[1165] Step 4:

[1166] The server uses a generative AI model to generate a custom model based on the received user customization requests and 3D model. The inputs are a prompt from the user and the 3D model. The server uses the generative AI model to execute an internal algorithm and generate a 3D model that matches the user's requests. This generated model is saved in .obj file format. Data processing is performed on the server side, and the progress is displayed.

[1167] Step 5:

[1168] The server sends the generated .obj file to the terminal. The input is the generated .obj file. The file is compressed and sent to the terminal over the network. The terminal receives the .obj file and notifies the user. As a result, the user's terminal displays "Custom model generated."

[1169] Step 6:

[1170] The user uses the terminal to check the generated .obj file. Using the 3D viewer, the appearance and dimensions of the model can be checked in detail. The input is the generated .obj file. The user checks and approves it. In operation, the 3D model is displayed on the screen and the user performs the check operation.

[1171] Step 7:

[1172] The user sends the .obj file from the device to the 3D printer, which is the output device. The input is the .obj file that the user has confirmed. The 3D printer generates the physical item based on this. As a result, the status of the 3D printer is displayed on the device screen, allowing the user to check the progress of the print. For example, the process of the user printing a smartphone holder is displayed in detail.

[1173] By following these steps, users can create custom items without any special expertise.

[1174] (Application example 1)

[1175] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1176] In modern factories, creating custom tools and parts to fit specific machine components is a complex and time-consuming task. Traditional methods require specialized knowledge and make it difficult to efficiently design and manufacture custom parts. This can lead to process interruptions and delays, reducing production efficiency, especially when rapid changes or modifications to production lines are required. To solve this problem, a system is needed that allows users to easily design and quickly manufacture custom parts.

[1177] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1178] In this invention, the server includes a sensor means for scanning an object, an information processing means for processing data acquired from the sensor means to generate a three-dimensional model, an input means for inputting a user's request for the generated three-dimensional model, a generation means for analyzing the request acquired from the input means and generating an appropriate data file based on the three-dimensional model, an output means for transmitting the generated data file to an output device to generate an object, and a control means implemented in a computer for controlling a machine tool appropriate for the generated data file, thereby enabling the rapid and efficient design and manufacturing of required custom tools and parts in a factory.

[1179] "Sensor means" refers to a device used to acquire information about the three-dimensional shape and depth of an object. Optical depth measurement devices and LiDAR sensors are mainly used.

[1180] "Information processing means" refers to a computer system that processes data acquired from the sensor means and generates a three-dimensional model. It analyzes, converts, and generates the data.

[1181] "Input means" refers to the interface through which the user inputs specific requests to the system. Chat interfaces and touch screens are typically used.

[1182] "Generation means" refers to a device or function that analyzes the 3D model and generates an appropriate data file based on the user's request input via the input means. The shape and size of the model are adjusted using an algorithm.

[1183] The "output means" is a device that generates the generated data file as a physical object. 3D printers and machine tools are usually used.

[1184] The "control means" is a system for controlling the machine tool based on the generated data file, and instructs the operation of the machine tool using a computer program.

[1185] Hardware Configuration

[1186] The system includes the following major hardware components:

[1187] 1. Sensor means: Optical depth measurement devices (such as LiDAR sensors) for obtaining three-dimensional shape and depth information of objects.

[1188] 2. Information processing means: A computer that processes the data obtained from the sensor means and generates a three-dimensional model.

[1189] 3. Input means: The interface through which users input their requests (e.g., a chat interface or touch screen on a smartphone).

[1190] 4. Generation means: A processing device that analyzes the three-dimensional model and generates a data file based on the user's request.

[1191] 5. Output means: A 3D printer or machine tool that produces a physical item based on the generated data file.

[1192] 6. Control means: A computer program that controls the machine tool based on the generated data file.

[1193] Software Structure and Algorithms

[1194] The system uses libraries such as open3d to process 3D point cloud data and generate .obj files, and also uses an AI model to analyze user input and generate an appropriate 3D model.

[1195] Operation procedures and data processing

[1196] The server processes the data and generates the custom items as follows:

[1197] 1. Data acquisition: Using sensor means to acquire depth information of the scanned object. For example, scanning a specific part in a factory with a smartphone's LiDAR sensor.

[1198] 2. 3D model generation: The acquired data is sent to the information processing means, and a 3D model is generated using the open3d library.

[1199] 3. Request input: The user uses the chat interface on their smartphone to input a specific request, for example, "I want to create an adapter that fits a pipe with a diameter of 5 cm and a length of 10 cm."

[1200] 4. Request analysis: The generator analyzes the input request and adjusts the 3D model. The shape and size are automatically adjusted using an AI model.

[1201] 5. Data file generation: The final adjusted 3D model is generated as a data file (.obj format).

[1202] 6. Output: The generated data file is sent to a 3D printer or machine tool to create the physical custom item.

[1203] 7. Machine Tool Control: A control means controls the machine tool based on the generated data file to rapidly manufacture precision custom parts.

[1204] Specific examples

[1205] For example, to create an adapter to fit a specific machine part in a factory, a user might do the following:

[1206] 1. Scan the target pipe with your smartphone's LiDAR sensor.

[1207] 2. Type into the chat interface, "I want to generate an adapter that fits a pipe that is 5cm in diameter and 10cm long."

[1208] 3. The server generates a 3D model based on the acquired data and adjusts it to suit your needs.

[1209] 4. The generated .obj file is sent to a 3D printer to produce the physical adapter.

[1210] Prompt Sentence Examples

[1211] "Create an .obj file that generates a custom adapter that fits a pipe that is 5cm in diameter and 10cm long."

[1212] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1213] Step 1:

[1214] A sensor means is used to obtain depth information of the scanned object.

[1215] Input: Object to scan

[1216] Data processing: A depth sensor (LiDAR sensor) scans the area around the object and obtains three-dimensional point cloud data.

[1217] Output: Acquired 3D point cloud data

[1218] Step 2:

[1219] The information processing means processes the acquired three-dimensional point cloud data to generate a three-dimensional model.

[1220] Input: 3D point cloud data

[1221] Data processing: Analyze the 3D point cloud data using the open3d library and generate a 3D model.

[1222] Output: Generated 3D model

[1223] Step 3:

[1224] The user uses the chat interface of the terminal to input a custom request.

[1225] Input: User's custom request (e.g. "I want to create an adapter that fits a pipe with a diameter of 5 cm and a length of 10 cm")

[1226] How it works: A user types a specific request into the chat interface on their smartphone or device.

[1227] Output: Input custom request

[1228] Step 4:

[1229] The generating means analyzes the input request and adjusts the three-dimensional model.

[1230] Input: Generated 3D model, custom request

[1231] Data calculation: Using a generative AI model, the shape and size of the 3D model are adjusted based on the input customization requests. Specifically, the system analyzes the user's requests and appropriately changes the shape parameters of the 3D model.

[1232] Output: Adjusted 3D model

[1233] Step 5:

[1234] Generate a suitable data file (.obj format) based on the adjusted 3D model.

[1235] Input: Calibrated 3D model

[1236] Data Generation: Export the 3D model in .obj file format.

[1237] Output: Generated .obj file

[1238] Step 6:

[1239] An output method is used to send the generated .obj file to a 3D printer or machine tool to manufacture the physical custom item.

[1240] Input: Generated .obj file

[1241] Data Transfer: Send the generated .obj file over the network to the 3D printer or machine tool.

[1242] How it works: The 3D printer deposits filament based on the .obj file to create a physical custom part.

[1243] Output: The generated physical custom item

[1244] Step 7:

[1245] The control means controls the machine tool based on the generated data file to manufacture precision custom parts.

[1246] Input: Generated .obj file

[1247] Control operation: The control program installed in the machine tool drives the machine tool based on the contents of the .obj file to manufacture a custom part with the specified shape.

[1248] Output: Precision-manufactured custom parts

[1249] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1250] The present invention provides a system that allows users to easily create items that perfectly fit into specific locations or objects. The system includes a depth sensor, a processing unit, an input unit, a generating unit, an output unit, and an emotion engine. These components work together to generate custom items based on the user's requests and emotion information.

[1251] First, a user scans an object using a depth sensor (e.g., an optical depth measurement device built into a smartphone). The user holds the smartphone and moves around the object to acquire depth information. This information is sent in real time to a processing unit, which then generates a three-dimensional model.

[1252] Next, the user uses a chat interface within the application on the device to input customization requests for the generated 3D model. For example, a specific request such as "I want a cup holder with a diameter of 10 cm and a height of 5 cm" can be input into the device. At the same time, the emotion engine analyzes the user's facial expressions and voice to obtain emotional information. This input information and emotional information are analyzed and sent to the generation means.

[1253] The generator analyzes the user's requests and emotional information and adjusts the shape and size of the 3D model data appropriately. For example, if the user is excited, bold elements are added to the design, and if the user is relaxed, the design is adjusted to be calm. A dedicated algorithm is used for this adjustment, and finally an .obj file that reflects the user's requests and emotions is generated.

[1254] The generated .obj file is displayed to the user via the terminal, and the user can confirm its contents. After confirmation, the user sends the generated data file from the terminal to the output means, i.e., the 3D printer, which generates a physical item based on the .obj file.

[1255] As a concrete example, let's say a user wants to create a pen holder that fits perfectly on their desk at home. In this case, the user uses the optical depth measurement device on their smartphone to scan the desired area of ​​the desk. Meanwhile, the emotion engine ensures that the user is relaxed. Next, they input in the application, "I want to create a pen holder that is 10cm tall and 5cm wide." The generator analyzes this and generates an .obj file of the pen holder that incorporates emotional information and calm design elements. Finally, the user can print this file on a 3D printer and place the finished pen holder on their desk.

[1256] In this way, the system of the present invention provides a means for users to easily create custom items they need in their daily lives, and can be used without any special skills. Furthermore, the introduction of an emotion engine enables customization according to the user's emotional state, providing a more satisfying user experience.

[1257] The processing flow will be explained below.

[1258] Step 1:

[1259] User: Launches the dedicated application on the smartphone and starts using the LiDAR sensor by pointing it at the object to be scanned. The user holds the smartphone and moves around the object to acquire depth information.

[1260] Step 2:

[1261] On the device: Depth data acquired by the LiDAR sensor is processed in real time to generate 3D point cloud data, which is then visually displayed in the application, allowing the user to follow the progress of the scan.

[1262] Step 3:

[1263] User: Once the scan is complete, press the "Scan Complete" button within the application to finalize the scan data.

[1264] Step 4:

[1265] Terminal: Compresses and converts the format of the confirmed point cloud data and sends it to a server via the Internet.

[1266] Step 5:

[1267] Server: Receives point cloud data sent from the device, verifies the data's normality and consistency, then generates a 3D mesh from the point cloud data and converts it into a format usable by the generation AI.

[1268] Step 6:

[1269] User: After completing the scan, the user accesses the chat interface within the application and enters their requirements and wishes for the item they wish to create, for example, "I would like to create a vase that is 10cm tall and 7cm in diameter."

[1270] Step 7:

[1271] Terminal: The text analysis engine processes the user's input, extracts the appropriate instruction information, and sends it to the server.

[1272] Step 8:

[1273] Server: Based on instructions from the text analysis engine, the generation AI adjusts the shape and size of the 3D mesh data and generates an .obj file that meets the requirements.

[1274] Step 9:

[1275] Server: Sends the generated .obj files to the device.

[1276] Step 10:

[1277] Terminal: The received .obj file is displayed to the user, providing a preview, allowing the user to review the details of the result and request changes or corrections if necessary.

[1278] Step 11:

[1279] User: If there is no problem with the .obj file, select the "3D Print" option.

[1280] Step 12:

[1281] Terminal: Convert the .obj file into a format suitable for a 3D printer (e.g. G-code) and send it to the 3D printer.

[1282] Step 13:

[1283] Device: Once the print setup is complete, start your 3D print.

[1284] Step 14:

[1285] User: Once the 3D printer has finished printing, they retrieve the physical item and place or use it where they want.

[1286] Step 15:

[1287] On the device: At the same time, the emotion engine acquires emotion information through facial expression recognition and voice analysis, which is then analyzed together with the input information in the chat interface.

[1288] Step 16:

[1289] Server: Reflects the user's emotional information obtained by the emotion engine and adjusts the shape and design when generating the data file. For example, if the user is relaxed, a calm design is suggested.

[1290] Step 17:

[1291] Terminal: The adjusted .obj file is displayed to the user again for final confirmation.

[1292] Step 18:

[1293] User: Once final confirmation is complete, send it to the 3D printer for final printing.

[1294] Through these steps, the system of the present invention is implemented to enable users to easily create and output custom items that fit perfectly into specific locations or objects, and can reflect the user's emotional information in the process.

[1295] Example 2

[1296] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1297] Conventional 3D model generation systems often require users to have detailed technical knowledge and skills to create custom items that fit specific shapes and sizes. Furthermore, they lack the ability to customize designs based on the user's emotional state, leaving the user experience open to improvement.

[1298] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1299] In this invention, the server includes depth sensor means for scanning an object, processing means for processing data acquired from the depth sensor means to generate a three-dimensional model, an emotion engine for analyzing a user's facial expressions and voice to acquire emotion information, and output means for transmitting the generated data file to an output device and generating an object. This allows users to easily create custom items without special skills and enables customization of designs based on emotion information.

[1300] "Depth sensor"

[1301] is a device for measuring the distance and shape of an object, and primarily uses optical technology to obtain depth information.

[1302] "Processing means"

[1303] is a computing device or software that uses acquired data to generate a three-dimensional model.

[1304] "Input means"

[1305] is an interface through which the user inputs customization requests, and through which the user's instructions are transmitted to the system.

[1306] "Emotion Engine"

[1307] is a technology or device for acquiring emotional information by analyzing a user's facial expressions and voice.

[1308] "Generation means"

[1309] is a technology or device that analyzes requests obtained from an input means and emotional information obtained from an emotion engine, and generates an appropriate data file based on a three-dimensional model.

[1310] Output means

[1311] is a device for sending the generated data file to an output device to generate a physical object.

[1312] "Three-dimensional model"

[1313] is a digital representation created based on the shape and size of an object, and refers to three-dimensional structure data composed of depth information.

[1314] "Emotional information"

[1315] This is psychological state data that is analyzed by the emotion engine from the user's facial expressions and voice.

[1316] Data File

[1317] is a file format in which the three-dimensional model data is output after being appropriately processed by the generating means.

[1318] This invention provides a system that allows users to easily create items that perfectly fit into specific locations or objects. The system includes a depth sensor, processing means, input means, an emotion engine, generation means, and output means. These components work together to generate custom items based on the user's requests and emotion information.

[1319] First, the user scans the object using a depth sensor (for example, an optical depth measurement device built into a smartphone). The user holds the smartphone and moves around the object, acquiring depth information. This information is sent in real time to a processing means, which generates a 3D model. Specifically, the smartphone application analyzes the depth data and updates the 3D model every 0.1 seconds. This model is displayed on the application's main screen, where the user can rotate and zoom the model.

[1320] Next, the user uses a chat interface within the application on the device to input customization requests for the generated 3D model. For example, a specific request such as "I want a cup holder with a diameter of 10 cm and a height of 5 cm" can be input into the device. At the same time, the emotion engine analyzes the user's facial expressions and voice to obtain emotional information. This analysis is performed using a camera and microphone. If the user is smiling or speaking in a calm voice, the emotion engine determines that the user is relaxed.

[1321] The desire and emotion information acquired from the input means is sent to the generation means. The generation means analyzes this information and adjusts the 3D model data to an appropriate shape and size. For example, if the user is relaxed, adjustments are made to incorporate calm design elements. This ultimately generates an .obj file that reflects the user's desire and emotion. A Python-based shape generation algorithm is used for this adjustment.

[1322] The generated .obj file is displayed to the user through the terminal, and the user can confirm its contents. After confirming, the user sends the generated data file from the terminal to the output means, i.e., the 3D printer. The 3D printer generates a physical item based on the .obj file.

[1323] As a concrete example, let's say a user wants to create a pen holder that fits perfectly on their desk at home. In this case, the user uses the optical depth measurement device on their smartphone to scan the desired area of ​​the desk. Meanwhile, the emotion engine confirms that the user is relaxed. Next, they input in the application, "I want to create a pen holder that is 10 cm tall and 5 cm wide." The generator analyzes this and generates an .obj file of the pen holder that incorporates calming design elements, incorporating the emotion information. Finally, the user can print this file on a 3D printer and place the finished pen holder on their desk.

[1324] An example prompt is:

[1325] "I would like to make a pen holder that is 10cm tall and 5cm wide. Please keep the design calm."

[1326] In this way, the system of the present invention provides a means for users to easily create custom items they need in their daily lives, and can be used without any special skills. Furthermore, the introduction of an emotion engine enables customization according to the user's emotional state, providing a more satisfying user experience.

[1327] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1328] A detailed explanation of the system's program divided into processing steps

[1329] Step 1: Acquiring and sending depth information

[1330] concrete action

[1331] The user launches the smartphone app and switches to scan mode. The device uses the smartphone's built-in optical depth measurement device to scan the area around an object designated by the user and collect depth information.

[1332] input

[1333] The user selects the object they want to scan and moves around while holding the smartphone.

[1334] Data processing and calculation

[1335] The device processes the acquired depth data in real time and generates three-dimensional coordinate data.

[1336] output

[1337] The processed three-dimensional coordinate data is generated and sent to the next step.

[1338] Step 2: Generate a 3D model

[1339] concrete action

[1340] The device receives the processed depth data in real time and generates a 3D model, which the user can view on the application screen and manipulate, such as rotating and zooming.

[1341] input

[1342] Three-dimensional coordinate data obtained by scanning.

[1343] Data processing and calculation

[1344] The processing means builds a model based on the 3D coordinate data, and updates the model every 0.1 seconds.

[1345] output

[1346] The generated three-dimensional model is displayed in real time on the user's terminal screen.

[1347] Step 3: Enter your customization request

[1348] concrete action

[1349] The user uses the chat interface within the application on the device to input customization requests for the generated 3D model. For example, the user might input, "I want to create a cup holder that is 10 cm in diameter and 5 cm in height."

[1350] input

[1351] Specific customization requests entered by the user into the chat interface.

[1352] Data processing and calculation

[1353] The user's request is analyzed as text and passed to the generating means as an appropriate parameter.

[1354] output

[1355] The desired parameters after text analysis are sent to the next step.

[1356] Step 4: Acquiring emotional information

[1357] concrete action

[1358] Using a camera and microphone, the emotion engine analyzes the user's facial expressions and tone of voice. For example, if the user is smiling or speaking in a calm voice, it determines that the user is relaxed.

[1359] input

[1360] The user's facial expression data and voice data.

[1361] Data processing and calculation

[1362] The emotion engine uses facial recognition and voice analysis algorithms to determine the user's emotional state.

[1363] output

[1364] The analyzed emotion information is sent to the next step.

[1365] Step 5: Analyze and adjust the information

[1366] concrete action

[1367] The terminal receives the user's customization request and emotional information, and the generation means analyzes this information and adjusts the 3D model appropriately. For example, if the user is relaxed, an .obj file incorporating calm design elements is generated.

[1368] input

[1369] User customization request parameters and emotional information.

[1370] Data processing and calculation

[1371] The generating means adjusts the shape and size of the three-dimensional model based on the shape generation algorithm and generates an .obj file.

[1372] output

[1373] The generated .obj file is sent to the next step.

[1374] Step 6: Check and output the results

[1375] concrete action

[1376] The generated .obj file is displayed on the terminal screen for the user to review. Once the user confirms the design, the file is sent to the 3D printer, which creates the physical item based on the .obj file.

[1377] input

[1378] The generated .obj file.

[1379] Data processing and calculation

[1380] The system uses file transfer protocol to send the .obj file to the 3D printer.

[1381] output

[1382] The physical item is produced by a 3D printer.

[1383] In this way, each processing step has specific inputs and outputs, and specific actions are taken to make the entire system function.

[1384] (Application example 2)

[1385] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1386] Modern factory manufacturing processes require the rapid creation of specialized tools and parts. However, traditional manufacturing methods require significant time and effort to design and create specialized tools, significantly reducing the efficiency of factory workers. Furthermore, existing systems struggle to automatically generate custom tools that reflect user feedback and emotions. As a result, the flexibility and efficiency of the manufacturing process are limited. To address these challenges, a new system is needed that can generate custom tools based on user requests and emotion information in a fast and automated manner.

[1387] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a depth sensor means for scanning an object, a processing means for processing data acquired from the depth sensor means to generate a three-dimensional model, an input means for inputting a user's desires and emotional information for the generated three-dimensional model, an output means for transmitting the generated data file to an output device and generating an object, and an industrial robot means for generating an automatically adjusted custom tool based on the analyzed desires and emotional information. This enables factory workers to quickly and efficiently create specialized tools. Furthermore, providing custom tools that reflect the user's emotional information significantly improves the flexibility and efficiency of the manufacturing process.

[1388] The "depth sensor means" is a device for detecting the depth and distance information of an object.

[1389] The "processing means" refers to a device or program for generating a three-dimensional model based on the acquired depth data.

[1390] "Input means" refers to a device or interface that allows a user to input requests and emotional information to a three-dimensional model.

[1391] "Generation means" refers to a device or program that analyzes the user's requests and emotional information and generates a data file based on a three-dimensional model.

[1392] An "output means" is a device or interface for converting the generated data file into a physical form.

[1393] An "industrial robotic tool" is a device such as a robotic arm for automatically creating custom tools or objects based on a generated data file.

[1394] "User requirements" are specific requests that users have for the system, such as the shape, size, and functionality of custom tools or objects.

[1395] "Emotion information" is data that indicates the emotional state of the user, obtained from facial expressions, voice, etc.

[1396] A "3D model" is three-dimensional digital data that represents the depth and shape of a scanned object.

[1397] The "data file" is the design data of a custom tool based on a three-dimensional model or user requirements, which is created by the generating means.

[1398] "Custom Tools" are specialized tools and parts designed and produced based on a user's specific requirements.

[1399] To implement this invention, a system is constructed in the following steps: First, a user scans an object using a depth sensor, which can be a smartphone or tablet equipped with an optical depth measurement device, and data on the object's depth and shape is acquired.

[1400] This depth data is then sent in real time to a processing unit within the device, which generates a three-dimensional model using 3D modeling software such as Blender. Users can also use input units within the smartphone application to enter requests. For example, they can use a chat interface to enter a specific request, such as "I want a wrench with a diameter of 5 cm."

[1401] Furthermore, the user's emotional information is acquired through the device's camera and microphone and analyzed by an emotion engine (e.g., Microsoft Azure Face API), allowing customization according to the user's emotional state.

[1402] The generation means adjusts the shape and size of the 3D model data appropriately based on the user's requests and emotional information. A dedicated algorithm analyzes this, and finally generates a data file in .obj file format. This data file is sent by the server to a robot arm (e.g., Universal Robots' UR5). The server instructs the robot arm to create a custom tool based on the generated data file.

[1403] Finally, the robotic arm creates a custom tool based on the generated .obj file, which is then physically printed using a 3D printer (e.g., Formlabs Form 3) as the output method, allowing users to quickly obtain the specialized tool they desire.

[1404] As a concrete example, imagine a factory worker needs a specific tool that perfectly fits the design of a new product line. In this case, the worker uses their smartphone to scan an existing part, and the emotion engine detects their emotional state as relaxed. They then type, "I need a special wrench that's 10cm tall and 5cm wide" into the application's chat interface. The generator analyzes this, automatically adjusts the design of the custom tool based on the emotion information, and generates an .obj file. A robotic arm then creates the tool based on this design, and finally, a 3D printer physically prints it. The worker can then use the tool immediately.

[1405] Here is an example of a prompt to input to a generative AI model:

[1406] Based on the invention below, design an application for a custom tool-making system for factory workers.

[1407] Scan objects using depth sensing means

[1408] The acquired depth data is processed in real time to generate a 3D model

[1409] Enter user requests using the chat interface

[1410] Acquire and analyze user emotional information using an emotion engine

[1411] Generate .obj files based on 3D model data

[1412] The generated data file is sent to the robot arm

[1413] Robotic arm automatically generates custom tools

[1414] Finally, the custom tool is printed on a 3D printer.

[1415] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1416] Step 1:

[1417] The user scans the object using the depth sensor means. The depth sensor means uses an optical depth measurement device on a smartphone or tablet to acquire data on the depth and shape of the object. The input is the object to be scanned, and the output is the depth data of the object. The user holds the smartphone and moves around the object to be scanned.

[1418] Step 2:

[1419] The acquired depth data is sent in real time to the processing means within the device. As the processing means, 3D modeling software (e.g. Blender) is used to generate a three-dimensional model based on the depth data. The input is depth data and the output is a three-dimensional model. The device analyzes the depth data and generates a three-dimensional model of the object.

[1420] Step 3:

[1421] The user uses a chat interface within the smartphone application to input their requests for the generated 3D model. Text input or voice input can be used as the input method. The input is the user's request (e.g., "I want a wrench with a diameter of 5 cm"), and the output is the request information. The user inputs a specific request.

[1422] Step 4:

[1423] The device acquires the user's emotional information through the camera and microphone and analyzes it using an emotion engine (e.g., Microsoft Azure Face API). The input is the user's facial expression and voice, and the output is the analyzed emotional information. The device determines the user's emotional state.

[1424] Step 5:

[1425] The generator adjusts the shape and size of the 3D model data appropriately based on the user's request and emotional information. A dedicated algorithm analyzes this and finally generates a data file in .obj file format. The input is request information and emotional information, and the output is the adjusted 3D model data and .obj file. The generator automatically adjusts the design.

[1426] Step 6:

[1427] The server sends the generated data file to an industrial robotic means (e.g., a robot arm). The input is an .obj file, and the output is the data file transfer to the robot arm. The server sends instructions to the robot arm to process the generated model.

[1428] Step 7:

[1429] The robot arm creates a custom tool based on the submitted .obj file. The input is the .obj file, and the output is the physical custom tool. The robot arm then machines the tool into the specified shape.

[1430] Step 8:

[1431] Optionally, the custom tool is physically printed with a 3D printer as the output medium. The input is an .obj file and the output is a 3D printed custom tool. The 3D printer generates the physical tool.

[1432] The above process provides a system that allows users to quickly and efficiently create specialized tools.

[1433] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1434] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1435] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1436] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1437] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1438] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1439] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1440] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1441] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1442] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1443] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1444] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1445] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1446] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1447] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1448] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1449] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1450] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1451] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1452] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1453] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1454] The following is further disclosed regarding the above embodiment.

[1455] (Claim 1)

[1456] a depth sensor means for scanning an object;

[1457] processing means for processing data acquired from the depth sensor means to generate a three-dimensional model;

[1458] an input means for inputting a user's request for the generated three-dimensional model;

[1459] a generating means for analyzing the request acquired from the input means and generating an appropriate data file based on the three-dimensional model;

[1460] output means for transmitting the generated data file to an output device to generate an object;

[1461] A system including:

[1462] (Claim 2)

[1463] 10. The system of claim 1, wherein the depth sensor means is an optical depth measurement device.

[1464] (Claim 3)

[1465] 10. The system of claim 1, wherein the generating means executes an algorithm that adjusts the shape and size of the three-dimensional model based on user requests.

[1466] "Example 1"

[1467] (Claim 1)

[1468] means for scanning a periphery of an object using a depth sensor means to obtain depth information;

[1469] a processing means for generating a three-dimensional model in real time based on the depth information and transmitting the model to a server;

[1470] an input means for providing a chat interface for a user to input customization requests;

[1471] A generation means for analyzing the user's customization requests, adjusting the 3D model using a generative AI model, and generating an optimal data file;

[1472] output means for returning the generated data file to the terminal, displaying it for the user to review, and finally transmitting it to an output device;

[1473] A system including:

[1474] (Claim 2)

[1475] 10. The system of claim 1, wherein the system uses an optical depth measurement device.

[1476] (Claim 3)

[1477] 2. The system of claim 1, wherein the generating means adjusts the shape and size of the three-dimensional model based on the user's request using a prompt sentence.

[1478] "Application Example 1"

[1479] (Claim 1)

[1480] sensor means for scanning an object;

[1481] an information processing means for processing the data acquired from the sensor means to generate a three-dimensional model;

[1482] an input means for inputting a user's request for the generated three-dimensional model;

[1483] a generating means for analyzing the request acquired from the input means and generating an appropriate data file based on the three-dimensional model;

[1484] output means for transmitting the generated data file to an output device to generate an object;

[1485] a control means implemented in a computer for controlling a machine tool suitable for the generated data file;

[1486] A system including:

[1487] (Claim 2)

[1488] 10. The system of claim 1, wherein the depth sensor means comprises a depth measuring device.

[1489] (Claim 3)

[1490] 10. The system of claim 1, wherein the generating means executes an algorithm that adjusts the shape and size of the three-dimensional model based on user requests.

[1491] "Example 2: Combining Emotion Engines"

[1492] (Claim 1)

[1493] a depth sensor means for scanning an object;

[1494] processing means for processing data acquired from the depth sensor means to generate a three-dimensional model;

[1495] an input means for inputting a user's request for the generated three-dimensional model;

[1496] an emotion engine for analyzing the user's facial expressions and voice to obtain emotional information;

[1497] a generating means for analyzing the request and emotion information acquired from the input means and generating an appropriate data file based on the three-dimensional model;

[1498] output means for transmitting the generated data file to an output device to generate an object;

[1499] A system including:

[1500] (Claim 2)

[1501] 10. The system of claim 1, wherein the depth sensor means is an optical depth measurement device.

[1502] (Claim 3)

[1503] 2. The system of claim 1, wherein the generating means executes an algorithm that adjusts the shape and size of the three-dimensional model based on the user's desires and emotional information.

[1504] "Application example 2 when combining emotion engines"

[1505] (Claim 1)

[1506] a depth sensor means for scanning an object;

[1507] processing means for processing data acquired from the depth sensor means to generate a three-dimensional model;

[1508] an input means for inputting user requests and emotional information for the generated three-dimensional model;

[1509] a generating means for analyzing the request and emotion information acquired from the input means and generating an appropriate data file based on the three-dimensional model;

[1510] output means for transmitting the generated data file to an output device to generate an object;

[1511] an industrial robotic means for generating an automatically adjusted custom tool based on the analyzed desire and emotion information;

[1512] A system including:

[1513] (Claim 2)

[1514] 2. The system of claim 1, wherein the depth sensor means uses an optical depth measurement device, and the generated data file is transmitted to a robot arm to generate the object.

[1515] (Claim 3)

[1516] 2. The system of claim 1, wherein the generating means executes an algorithm that adjusts the shape and size of the three-dimensional model based on the user's requests and emotional information, and applies the generated data file to an automated manufacturing process. [Explanation of symbols]

[1517] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a depth sensor means for scanning an object; processing means for processing data acquired from the depth sensor means to generate a three-dimensional model; an input means for inputting a user's request for the generated three-dimensional model; a generating means for analyzing the request acquired from the input means and generating an appropriate data file based on the three-dimensional model; output means for transmitting the generated data file to an output device to generate an object; A system including:

2. 10. The system of claim 1, wherein the depth sensor means is an optical depth measuring device.

3. 10. The system of claim 1, wherein the generating means executes an algorithm that adjusts the shape and size of the three-dimensional model based on user requests.

Images